Towards Scheduling Federated Deep Learning using Meta-Gradients for Inter-Hospital Learning
Authors: Rasheed El-Bouri
Tingting Zhu
David A. Clifton
Venue: ICLR 2022 Submitted (2022)
Keywords: federated learning
hospital

Abstract: Given the abundance and ease of access of personal data today, individual privacy has become of paramount importance, particularly in the healthcare domain. In this work, we aim to utilise patient data extracted from multiple hospital data centres to train a machine learning model without sacrificing patient privacy. We develop a scheduling algorithm in conjunction with a student-teacher algorithm that is deployed in a federated manner. This allows a central model to learn from batches of data at each federal node. The teacher acts between data centres to update the main task (student) algorithm using the data that is stored in the various data centres. We show that the scheduler, trained using meta-gradients, can effectively organise training and as a result train a machine learning model on a diverse dataset without needing explicit access to the patient data. We achieve state-of-the-art performance and show how our method overcomes some of the problems faced in the federated learning such as node poisoning. We further show how the scheduler can be used as a mechanism for transfer learning, allowing different teachers to work together in training a student for state-of-the-art performance.

Under review as a conference paper at ICLR 2022
TOWARDS SCHEDULING FEDERATED DEEP LEARN- ING USING META-GRADIENTS FOR HOSER-HOSPITAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Given the abundance and ease of access of personal data today, individual privacy has become of paramount importance, particularly in the healthcare domain. In this work, we aim to utilise patient data extracted from multiple hospital data centres to train a machine learning model without sacrificing patient privacy. We develop scheduling algorithm in conjunction with student-teacher algorithm that is deployed in federated manner. This allows a central model to learn from batches of data each federal node. The teacher acts between data centres update the main task (student) algorithm using the data that is stored in the various data centres. We show that the scheduler, trained using vetaaggaddeens, can effectively organise training and as a result train a machine learning model on a diverse dataset without needing explicit access to the patient data. We achieve otate-of-the-art performance and show how our method overcomes some of the problems faced in the federated learning such node poisoning. We further show how the scheduler can be used as mechanism for transfer learning, allowing different teachers to work together in training student for state-of-the-art performance.
INTRODUCTION
Federated learning a field that has emerged recently due to the abundance of data available today and the risks that this poses to individuals. Privacy (particularly of personal information) is of great importance and should be protected by researchers working in machine learning. In parallel, the emergence of electronic health records (EHRs) has allowed the digitisation of much personal information pertaining to the health conditions of individuals. EHRs are often used in many machine learning research projects (Shillaja al. (018)). This often involves the transfer and storage of very sensitive information which increases the risk of data leakage. As result, federated learning can minimise this risk by utilising the data at it's source rather than transferring it the researchers servers to be processed.
Machine learning researchers working with EHR data will be all familiar with the difficulty of gaining access to this information in the first place. It can be a very lengthy and exhausting process (for good reason) to gain access and utilise the EHRs of one healthcare institution let alone accessing the datasets of many. Federated learning offers an alternative in that it allows the data of the patients to be utilised while reducing the risk of their privacy being compromised.
Federated learning is not without its own limitations however. The different datasets that are stored in the different nodes may have different underlying distributions due to their data collection processes which can make machine learning across multiple domains difficult. There is also the possibility of data at each node being corrupted either maliciously or accidentally, leading to data that is undesirable to use for training. These issues can all lead to difficulties in training and convergence of the overall model being trained.
To overcome these limitations, in this work propose the following setup. Firstly, We use federated learning to i) protect the privacy of patients by minimising the movement their data and ii) improve the quality of our machine learning model by utilising diverse dataset sourced from different

---

Under review as a conference paper at ICLR 2022
hospitals, As are 'blind' to the data at the nodes, we propose the of student- network setup. The teacher (a reinforcement learning agent) will have access to the local servers and be able to select the appropriate data for the 'student' (our model) to be trained on at that time. The 'scheduler' will be responsible for directing the teacher to a given data centre to select data for training on.
To summarise: (The Student:) the machine learning model reinforcement learning agent that selects data from a data centre based on the state the student. This essentially defines a curriculum each training step for the student. (The Scheduler:) directs the teacher to the appropriate data centre for training. This is also based on the state of the student at each training iteration.
training. (The Teacher:)
Section 2 discusses the related work that has been carried out and Section 3 provides a detailed description of how our algorithm works. Section Partion4detann details the datasets we benchmark our method against. We then present the results in Section S and discuss their significance and interesting behaviours of our model in Section
2 RELATED WORK
Federated learning has been used by researchers to exploit larger pools of data for training Bonawitz ct al. (2019), preserve data privacy Xu et al. (2019) and distributing computational resource require- ments Yuan & Ma (2020). Federated learning has also been used for healthcare applications to simultaneously utilise multiple datasets to train a model on patient data.
In this work create model that learns in federal fashion through the interaction of scheduler that is trained using meta-gradients and a student-teacher algorithm that is trained using reinforcement learning.
Meta-learning has been used effectively in Such et al. (019)) where the loss of a student model on validation set was used as signal to update the weights of a generative model. This work demonstrated the rapid and effective training of methods that exploit meta meta-gradients. Meta-lcarning was also used in Zahavy et al. (2000) where the peta-ggaddents are used to tune the parameters of an actor-critic algorithm. As a result of the efficacy of this method in these domains we choose to use meta-grade geta-graddents order schedule which data centre the gradients update are student model will come from.
The meta-learned scheduler chooses a node representing a data centre where student-teacher algorithm is used to sample data. Student-teacher algorithms have been used in multiple works, with the general premise that one algorithm (teacher) is trained to train another (student) Fan et al. (2018);
Liu et al, (2017a). These methods have also been used with curriculum [Bengio et al. (2009) where the curriculum is either pre-defined and exploited by the teacher El-Bouri et al. (020)) or implicitly learned by the teacher during training Graves al. (0177).
Federated learning is a method of training a model (in our case a deep neural network) by using data from multiple centres, without having central access to each of them McMahan al. (2017). Local models at each of the data centres are iteratively updated and aggregated to form a global model. At each round of iteration, central coordinator samples a subset, m, of local models, Sm, and sends them the current global model Gt. Each member of S,, then updates this global model using their local data to create an updated model ++1, These models are then aggregated and are sent back to update the global model as:
m + [++1 1++1-69 -G¹)
GL+| G
n
where n the number nodes (i.e., data centres) and 7) acts as a learning rate for replacing the global model with the aggregate of the local models. While this has been shown to work in many cases, Wang et al. (2020a) make the argument that there inherent difficulty in updating neural networks in this manner. They argue that the permutation invariance of the summation operand renders averaging in the parameter space naive approach. For meaningful averaging to be done, the permutation must first be undone.
2

---

Under review as a conference paper at ICLR 2022
2.1 COMPROMISING FEDERATED LEARNING
One of the vulnerabilities of federated learning is that nodes being compromised can significantly affect the training of the global model Bhagoji al. (2018). Attacks of these sort can either 'poison' the data found at one of the nodes (known as an adversarial attack) or bias the model that is trained one of the nodes significantly, leading to highly skew the aggregation step Tolpegin et al. (2020). There is also the possibility of the attack being a single-shot attack or a repeated attack Fang et al. (2020). In the single shot case, only one of the nodes is compromised whereas in the repeated case, multiple nodes can be compromised at any given time. Many works have been produced in discussing how federated learning can compromised by introducing backdoor into the training process Gu al. (2017); Bagdasaryan et al. 22220. A backdoor is an attack that causes classifier to produce unexpected behaviour a specific trigger added to the input space. An example is a sticker being added to an image and associating this with the incorrect label Gu et al. (2017).
Defences against these attacks have been developed with some authors using pruning of redundant neurons for the core classification task Liu et al. (2018), using outlier detection to detect potential triggers Wang et al. (2019). and re-training and preprocessing inputs Liu et al. (017b).
In this work aim overcome these limitations and build defence into the training procedure through the use of suudett-teacher network that actively selects which data to train on.
3 METHODOLOGY
Our method is comprised of three agents in the training setup, the student, the teacher and the scheduler.
3.1 THE OVERALL SETUP
The overall setup of our federated learning training routine is as follows. We have scheduler that controls which node we will learning from (this can one-hot or we can select multiple nodes). The teacher at the node can then select batch of data according to the state of the student. The student at the node is a copy of the global student. We use the student to forward pass the batch of data selected by the teacher and return the loss. In the one-hot scheduler scenario, the loss to the global student model to update the weights via backpropagation. In the multi-node learning scenario, we aggregate the losses from all nodes selected and feed these back the global model for updating.
send back
3.2 DATA PREPROCESSING
The first step we must take order to exploit our teacher setup is to rank our data according to some metric. Using Wang et al. (2020b) guide, we choose to use the Mahalanobis distance expressed as:
d (x..) = ((xm -1)" ((xn-14** s-''sssn (m)--
for medical datasets, and the cosine similarity as our similarity metric for image datasets.
As the tabular data found in electronic health record systems consist of multiple data types, encode these using a denoising autoencoder. This trained encoder is distributed to all the nodes so that the data in each node is processed in the same way for consistency.
3.3 THE TEACHER
For the student-teacher interaction we follow the setup in El-Bouri et al. (2020). The task of the teacher is to select a batch of data from the curriculum by selecting the index along the curriculum and the 'width' around that index to include in the selection. The following sequential steps are implemented:
The data at each node is organised into N curriculum batches according some metric H. batches for feeding into the student.
The teacher selects
3

---

Under review as a conference paper at ICLR 2022
A pre-trained autoencoder is used create latent representation of the batch.
The student is trained on this batch and it's performance on a separate validation set is recorded.
In this work use the Mahalanobis distance for H for medical data, and cosine similarity for image data as summarised in Wang et al. (2020b).
The teacher is reinforcement learning agent and therefore is tasked with minimising the Bellman loss function given by:
[(,)) (r /
max (7 (3.
where is the reward state-action (s, a) tuple, y the discount factor, Q is the q-value defining the value of taking an action given a state and 0 and A are the parameters of the prediction and target (the version of the teacher that is held constant for K steps to stabilise training as described in Mnih ct al. 22015)) networks respectively.
As we also choose to use an actor-critic setup for the teacher, the action space and Q-function are separately parameterised. This allows continuous action space and the actor that selects actions is updated using the following loss:
1 N Ev.0(s.1109) (=s,,=p(ss) (s
VO
where Q is the Q-function and is the policy.
The teacher can either be pre-trained, jointly trained with the scheduler. The teacher also either be trained on the dataset of one node and distributed to the rest or independently trained at each node. The latter preferable due to the ability of the teacher adapt the dataset hand. However, the former is useful when not all nodes in the federated system have access to computational power. The intuition is that the curriculum strategy learned by the teacher should be general for the task at hand and thereby provide strong performance.
3.4 THE STUDENT
The input to the teacher is the current state the student. The student this work a feedforward neural network that is tasked with classification. The state of the student is defined representation of the weights of the student. Given a matrix of weights, Wii, between layers and j of the network, for each row, Wiy, we take the inner product of the row with fixed reference vector From this inner product we extract (W n a)l and 7 (W1) n: a) for n 1, 2,2.... M, where M; is the number of hidden nodes in layer i. These values are concatenated represent the row and this process is repeated for all rows to build the vector. For more hidden layers the process is repeated until we have network. This provides us with a representative vector, v RE M.), where h is the number of hidden layers. This vector is what is fed to the teacher to understand the state the student.
one vector representing
3.5 THE SCHEDULER
The scheduler the last of agents in the training setup. This also neural network that takes the student state as input and selects which of the nodes the training data should come from at the current iteration of training. This agent is trained using the grediegrs generated from validation losses similarly to how they are employed in Such et al. (2019). We have an 'inner' loop of training whereby the student-teacher interaction takes place. In the "outer" loop, we aggregate the losses on validation sets at each node and use this aggregated loss as the signal to update the weights of our scheduler. Figure [ shows diagrammatically the training procedure at every iteration of training. In the outer loop the scheduler selects the node(s) to use for data selection. The global student model sent to the node and the teacher at the node then selects the data the inner loop and trains the student network on this. The student is then sent back to the central node and distributed to all nodes. This student is tested on separate validation sets at each node and their losses are aggregated by summing them. They are then used as the loss to update the scheduler. The inner loop loss function
4

---

Under review as a conference paper at ICLR 2022
Outerioop
Updated
neer Loop
txsed
from processed data based
trainedon
stadent
pite
Agorogate
Mito #ignato polnt
achidd wragight
Figure I: A diagram displaying what happens in the training routine every iteration. The black arrows indicate selection, the blue indicate the extraction of state, dashed indicate the transfer of a model and the orange indicate the movement of losses.
is dependent upon the target task with crossentropy used for classification and mean squared error used for regression. The scheduler updated as:
g
fr + w
Σ OSPISE
a v (Cinner)
(5)
where Linner is given by:
N
Linner (T (s1; (i;
Σ Cinner (x", y Ose)
where 2se are scheduler weights, w is learning rate for stochastic gradient descent, a is a momentum hyperparameter, finner student state at iteration i, st. y is the ground truth associated with the teacher selection and 0.. and Osr the teacher and student parameterisations respectively. x and y are the features and labels of the validation set of node respectively.
local task loss function, T is the teacher network taking input the
ENTROPY Loss
As the scheduler is trained using meta-gradients, the model may converge after a set number of iterations. While would like convergence of the student (i.e., the task solving model), do not necessarily need the scheduler to converge. In fact using the loss on the validation set as the signal to update the scheduler automatically prevents the scheduler from converging for very long. This is because, should the scheduler converge on selecting particular data centre, the validation scores on the other data centres will deteriorate thereby increasing the aggregated loss the validations and providing an update signal to the scheduler. This however was found in practice to require many iterations of training and in order encourage the exploration further we add an entropic loss term to the scheduler. This takes the form:
Lent
H (8t5 0tc)l
where is entropy, 5 (8x; (%).) is the softmax output of the scheduler and is small positive value (we use 10-5) prevent potential division by zero. lent is then added to the end of the expression shown in Equation to discourage fast convergence of the scheduler.
For tabular classifications, define the student to be feedforward neural network consisting of 2 hidden layers and nodes each layer. These are all activated with ReLU activations. For the image recognition tasks initialise student that has convolutional layers with 32 filters of size 3x3 in the first two layers and 64 of these in the second two. These are all activated by ReLU and maxpooled and are followed by feedforward layers of size 50 nodes each.
For the results reported in Section 5| we use a pre- are-trained teacher (i.c.. the teacher has been trained using reinforcement learning different students for the same classification problem). The teacher
5

---

Under review as a conference paper at ICLR 2022
has 2 hidden layers and 150 nodes each activated by ReLU apart from the final layer (of size 2) which activated by a tanh function. For the scheduler we use a feedforward neural network with 2 hidden layers and 100 nodes in each layer activated by ReLU. The output is activated by a softmax of size the number of nodes in the federated system.
4 DATASETS
In this study we considered the patient data collected in the electronic health records (EHR) of the Other Unknown Hospital (OUH), which the authors are associated with. The data is split randomly in N datasets so that each can act separate machine in a federated system. The features include demographic, physiological and medical information (such as age, heart rate upon entry and any medical tests requested by clinical staff who greet the patient). We aim predict which department in the hospital the patient will consume resource from (i.e., which department in the hospital will ultimately be responsible for treating the patient) rendering this a seven-class classification. In carrying out this classification, this allows hospitals to predict their resource requirements ahead of time and update scheduling and planning accordingly. Only patients who were admitted in an emergency were considered providing dataset of 14,324 patients. A training set of 60% of the dataset was used and was balanced, leaving 8,589 patients for training on. The validation set was 20% of the dataset and testing was also 20%. These sets are then evenly divided according to the number of nodes in the federated system. The full feature set is included in the supplementary material.
EICU
In order to validate our results on real-world data collected from different hospitals, we introduce the eICU dataset Pollard et al. (2018) also hosted Physionet Goldberger Glylionet al. (2000). The task here is mortality prediction (binary classification) based on features extracted from admission to the ICU as done in SSeeikhhalisaai we are able to create nodes corresponding to each hospital. The features selected are as outlined in the appendix. We choose to learn from the eight hospitals with the largest populations in the dataset leaving us with 8,594 instances. We sample 60% from each node to keep as the training set, and keep 20% for the validation set and the final 20% as the test set. As per usual, the validation set is kept on the local node for performance aggregation during the scheduler training.
al. (2020). As this dataset contains identifiers for individual hospitals,
CIFAR-10
To test methodology on the image space we also report results on the CIFAR-10 10 image recognition dataset Krizhevsky et al. (2009). We use 40,000 training examples for the training set and 10,000 each for validation and to mimic a federated learning system. It should be noted that in Table the CIFAR-10 dataset has been split into three samples of size 15000, 15000 and 10000 due to the possibility of the teacher selecting full training set batchsize and memory constraints.
examples. We once again randomly divide the dataset into N datasets
MNIST
As before, we utilise the MNIST dataset LeCun et al. (1998) as another publicly available dataset to assess our results against. We use 30,000 examples for training, 10,000 for validation and 10,000 for the test set. Once again the dataset is partitioned into N datasets to emulate the federated learning approach.
RESULTS
As our work lies in the intersection of two research areas within the field of machine learning (namely federated learning and student-teacher learning), we choose to use baselines from both of these fields comparators. From the student-teacher learning side, we will assess how method compares in terms of final model performance only. For the federated learning comparison we will compare not only the final model performance but also the robustness of the method to attack.

---

Under review as a conference paper at ICLR 2022
Table 1: Average classification accuracies and standard deviations for various baseline and state-of- tre-art methods on the Ward Admission (tabular), MNIST (image) and CIFAR-10 10 (image) datasets. All models are averaged over the same five seeds apart from those highlighted with * which indicates that the accuracy reported from the cited text is quoted.
WARD ADMISSION
MNIST
CIFAR-10
CIFAR-10
CIFAR-10
EICU
SAMPLE
SAMPLE 2
SAMPLE
METHOD
Acc (SD)
Acc (SD)
Acc (SD)
AcC (SD)
Acc (SD)
AUC (SD)
SMBT
0.45 (0.01)
0.91 (0.01)
0.65 (0.02)
0.65 (0.02)
0.65 (0.02)
0.80 (0.02)
CURRIC
0.48 (0.02)
0.93 (0.02)
0.68 (0.01)
0.68 (0.01)
0.68 (0.01)
0.81 (0.01)
DEEPFM
0.59 (0.01)
N/A
N/A
N/A
N/A
0.81 (0.01)
DEEP4CROSSNET
0.58 (0.02)
N/A
N/A
N/A
N/A
0.82 (0.02)
DENSENET*
N/A
0.99 (0.01)
0.96 (0.01)
0.96 (0.01)
0.96 (0.01)
N/A
GPIPE*
N/A
0.99 (0.01)
0.99 (0.01)
0.99 (0.01)
0.99 (0.01)
N/A
RLST
0.62 (0.02)
0.95 (0.02)
0.90 (0.01)
0.90 (0.01)
0.89 (0.01)
0.85 (0.02)
FLST
0.60 (0.02)
0.94 (0.01)
0.91 (0.01)
0.90 (0.01)
0.91 (0.02)
0.86 (0.01)
Table 2: Average classification accuracies and standard deviations for various baseline and state-of- the-art methods the hospital ward admission (tabular), eICU (tabular), CIFAR-IO (image) and MNIST (image) datasets. All models are averaged over the same five seeds. We also show how the models perform when subjected to data poisoning attack and local model poisoning attack.
ACCURACY Acc (SD)
MODEL POISONING
DATA POISONING
METHOD
Acc (SD)
Acc (SD)
FEDAVG (HOSPITAL)
0.55 (0.01)
0.33 (0.01)
0.47 (0.02)
FEDMA (HOSPITAL)
0.56 (0.02)
0.45 (0.02)
0.52 (0.01)
FLST (HOSPITAL)
0.60 (0.02)
0.59 (0.01)
0.59 (0.02)
FEDAVG (CIFAR-10)
0.9300001)
0.62 (0.01)
0.65 (0.02)
FEDMA (CIFAR-10) 10)
(0.02)
0.77 (0.02)
0,68 (0.01)
FLST (CIFAR-10)
(0.01)
0.85 (0.01)
0.87 (0.02)
FEDAVG (MNIST)
(0.01)
0.77 (0.01)
0.73 (0.02)
FEDMA (MNIST)
0.95 (0.01)
0.83 (0.01)
0.81 (0.02)
FLST (MNIST)
0.95 (0.02)
0.86 (0.02)
0.88 (0.01)
FEDAVG AUC (EICU)
0.80 (0.01)
0.65 (0.07)
0.69 (0.12)
FEDMA AUC (RICU)
0.82 (0.01)
0.71 (0.09)
0.72 (0.04)
FLST AUC (EICU)
0.86 (0.02)
0.80 (0.04)
0.81 (0.03)
FINAL MODEL. PERFORMANCE
Table shows how the performance of our federated learning method (FLST) compares to other state-of-ther tth-aat classification methods. The baselines
the reinforcement learning trained student-teacher setup without scheduling El-Bouri et al. (2020) (RLST), two state-of-the-art methods used for classifying tabular data (DeepFM Guo al. (2017) and Deep+CrossNel Wang al. (2017) and two state-of the-arrttt classifiers for image recognition (GPipe Huang et al. (219)) and DenseNet Huang et al. (2018). ). As baselines train standard feedforward neural network (a convolutional neural network for the image datasets) using stochastic mini-batch training (SMBT) and a curriculum (CURRIC) for comparison.
We see that our federated system is capable of producing a performance that is competing with state-of-the-art models that are trained in centralised manner.
ROBUSTNESS OF FEDERATED TRAINING ROUTINE
Table 2 shows how our method performs when compared to other federated learning algorithms. For our baselines we use FedAvg McMahan et al. (2017) where the local models at each node are aggreegated before being averaged, well as FedMA Wang et al. (2020a), which constructs shared
7

---

Under review as a conference paper at ICLR 2022
global model in layer- layer-wise manner by matching and averaging hidden elements (such as neurons and hidden states). We investigate how performance deteriorates when exposed to different backdoor attacks. We see that our model performs equivalently to state-of-the art federated training setups in terms of test-time performance but outperforms these models when exposed to attack. Through the of the scheduler, our approach provides added layer of redundancy in the system thereby allowing attacks to be avoided after their implicit detection through degraded performance on the validation sets stored all nodes.
IMPLICIT DEFENSIVE SETUP
There are various ways in which federated system can be attacked. This can be through the poisoning of local models at each of the nodes or poisoning of the data at each of the nodes. In the supplementary material, we outline experiments that show that the scheduler chooses appropriate nodes for training. We show how using teaching setup, we can avoid some of the issues that could be faced by a federated learning system under attack.
FEDERATED LEARNING FOR A LIVE HOSPITAL ENVIRONMENT
To further assess the robustness of the proposed method, experiment treating the hospital live environment is used. This is important because randomly splitting the data for each node may not be entirely characteristic of how real-world data from multiple hospital data centres would distributed. There can be systematic differences due to the equipment that is used to take measurements, or the culture of measurement at the hospital, as well as downtime of machines meaning that recording is not possible. For this experiment, the data at two nodes are 'poisoned' in a systematic way that may mimic how data could be recorded a real hospital environment. To achieve this, the data are separated into four nodes according the time period during which the patient was admitted to the ED. These four nodes are patients admitted: (i) between midnight and 6 a.m., (ii) between a.m. and noon, (iii) between mimics the scenario where during the morning hours, historic patient records are not available and the historic diagnosis code is therefore replaced by the generic historic diagnosis code corresponding to headache, Figure 2 shows the scheduler's ' selections for this setup. We observe that the scheduler
and p.m. and (iv) between p.m. and midnight. The simulation
hosp1
hosp2
hosp3
0.8
hosp4
Softmax output
0.6
0.2
1000
2000
3000
4000
5000
Iteration
Figure 2: Scheduler selection in system where data has been systematically poisoned according to the time it was recorded in order to simulate database outage. Hosp (node i) and 2 (node ii) represent the patients admitted in the morning hours, and 3 (node iii) and 4 (node iv) the afternoon and evening admissions respectively.
begins by using the data from the morning but rapidly changes using afternoon data with which it
8

---

Under review as a conference paper at ICLR 2022
has success and therefore continues to train with. There is then cycling between the use of afternoon and evening data while neglecting to use any data recorded during the morning. Once again, this performance is conditional
the validation set at each node being secure and unchanged. As result, this indicates that some curation is required by a user for the validation sets to ensure that these are accurate. However, even with less curation of the training set, the scheduler may still be allowed to choose the centres from which it will gain greatest benefit for the student.
8 DISCUSSION AND CONCLUSION
In this work we have shown that using federated system, with teachers at the nodes that select the data to provide gradient updates central model, can achieve state-of-the-art performance well as protect the privacy of patients. We have further shown that the setup provides some protection against attacks on data stored at each node or the local models being used at each node.
However, there remain some challenges associated with this approach that need to be addressed in order for become practicable. The first that the training of the centralised model (student) inherently unstable due to the continual training that this setup expects. It is expected that the student performance will converge with training. however should data poisoning occur, the scheduler will need to continue few iterations of training in order to recover well-performing model. However, posing the problem this way also provides flexibility for growing datasets at each node, All that would need to happen would be the re-sorting of the curriculum at each node and the scheduler and teachers could be used as before.
Another limitation is the need for centralised control. It is important in this setup that all the hospitals communicate their responses to the central node for actions to be taken by the scheduler. In the case of large institutions such hospitals, this may be acceptable, but is unlikely to be for faster-paced learning environments such as learning from mobile phones, where interruptions to communication be frequent. However, upon re-connection the federated system, any reductions performance to the whole system will be used as signals to improve the scheduler selection and with training the performance should recover.
Furthermore, in order to ensure diversity in selections by the scheduler we introduced the entropy loss that discouraged convergence on one selection. This may be naive way of encouraging diversity in selection and we believe that there may be better additive losses and regularisation terms that can be used design loss function that will serve the purposes of the scheduler better.
For further protection against attack, sentry agents (much like the teachers) could also be trained to detect any anomalies or designed attacks within the batches selected by the teachers before the losses are passed onto the central node. This would reduce the burden of scanning the entire dataset at the node before training.
To conclude, we believe we have presented promising direction for federated learning between large institutions such as hospitals. With further work, we believe that we can develop this into robust system that continually learn from growing datasets while maintaining ofate-of-the-- performance for the task at hand.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-2948. PMLR, 2020.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Model poisoning 32nd Conf. Neural
attacks in federated learning. In Proc. Workshop Secur. Mach. Inf. Process. (NeurlIP)) 2018.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federated learning at scale: System design. arXiv preprint arXiv:1902.01046, 2019.
Rasheed El-Bouri. David Eyre, Peter Watkinson, Tingting Zhu, and David Clifton. Student-teacher
curriculum learning via reinforcement learning: predicting hospital inpatient admission location. arXiv preprint ar 20v::
2020.
Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and Tie- Tie-Yan Liu. Learning teach. arXiv preprint 2018.
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to byzantine-robust federated learning. In 29th {USENIX) Security Symposium ([USENIX) Security 20), pp. 1605-1622, 2020.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph Mietus, George B Moody, Chung-Kang Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23):e215-2222 2000.
Alex Graves, Mare Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated 2017.
curriculum learning for neural networks. arXiv preprint
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint
2017.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-
machine based neural network for etr prediction. arXiv preprint 1tXiv:1703.04244 20177
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2752-2761 2018.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pp. 103-112, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 11):228-22324 1998.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Antaks, Intrusions, and Defenses, pp. 273 273-294. Springer, 2018.
Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg. and Le Song. Iterative machine teaching. arXiv preprint arXiv: 10470, 2017a
Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In 2017 IEEE International Conference Computer Design (ICCD), pp. 45-48. IEEE, 2017b.
10

---

Under review as a conference paper at ICLR 2022
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282, PMLR, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
Tom Pollard, Alistair EW Johnson, Jesse Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):1-13, 2018.
K Shailaja, B Seetharamulu, and MA Jabbar. Machine learning in healthcare: A review. In 2018 Second international conference electronics, communication and aerospace technology (ICECA), pp. 910-914. IEEE, 2018.
Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani. Benchmarking machine learning models on multi-centre eicu critical care dataset. Plos one, 15(7):e0235424, 2020.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. arXiv preprint arXiv:1912.07766 2019.
Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. Data poisoning attacks against federated learning systems. In European Symposium on Research in Computer Security, pp. 480-501. Springer, 2020.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath. Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. 2019
IEEF Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.064400 2020a.
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions.
In Proceedings of the ADKDD'I1 17, pp. 1-722017
Xin Wang, Yudong Chen, and Wenwu Zhu. comprehensive survey on curriculum learning. arXiv preprint arXiv:2010. 10n::2000..11600 13166, 2020b.
Runhua Xu, Nathalic Baracaldo, Yi Zhou, Ali Anwar, and Heiko Ludwig. Hybridalpha: An efficient approach for Seavaccyprreeervvie federated learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, pp. 13-23, 2019.
Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. arXiv preprint arXiv:2006.08950, 2020.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junbyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. self-tuning actor-critic algorithm. Advances in Neural Information Processing Systems, 33, 2020.


Reviews:
[{"review_id": "Li5_FdpVi0K", "reviewer": "Reviewer_waUP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work applies federated learning with a neural scheduler to improve the effectiveness and robustness of the global model.  The idea is interesting, but they're still several concerns based on the current submission:\n1. Adopt deep reinforcement learning (DRL) as a neural scheduler is not new.  In [1], the authors also adopt DRL technique to filter the data or gradient boost the training speed and performance. It seems that the main contribution of this work is to extend the DRL-based neural scheduler to the federated learning setting, which limits the novelty of the proposed work. Hence, we recommend the authors give more related works and detailed comparisons on this submission with existed (DRL-based) neural scheduler. \n2. The experiments are not sufficient to demonstrate the efficacy of the proposed algorithm. In fact, there exist a large amount of work that uses hypernetwork [2] or bandit [3,4,5, etc] techniques to perform client selection.  Hence, we recommend the authors give baselines to demonstrate the efficacy of the proposed algorithm.\n3. The experients demonstrate the proposed algorithm is robust to backdoor attacks. This phenomenon is interesting. However, the reason why this approach works is still unknown. We recommend the author gives more explanation on these observations either in theory or in practice. \n\n[1]  Fan Y, Tian F, Qin T, Bian J, Liu TY. Learning what data to learn. arXiv preprint arXiv:1702.08635. 2017 Feb 28.\n[2] Shamsian, A., Navon, A., Fetaya, E. and Chechik, G., 2021. Personalized Federated Learning using Hypernetworks. arXiv preprint arXiv:2103.04628.\n[3] Xia W, Quek TQ, Guo K, Wen W, Yang HH, Zhu H. Multi-armed bandit-based client scheduling for federated learning. IEEE Transactions on Wireless Communications. 2020 Jul 16;19(11):7108-23.\n[4] Huang T, Lin W, Shen L, Li K, Zomaya AY. Stochastic Client Selection for Federated Learning with Volatile Clients. arXiv preprint arXiv:2011.08756. 2020 Nov 17.\n[5] Yoshida, N., Nishio, T., Morikura, M. and Yamamoto, K., 2020, December. MAB-based Client Selection for Federated Learning with Uncertain Resources in Mobile Networks. In 2020 IEEE Globecom Workshops (GC Wkshps (pp. 1-6). IEEE.\n", "summary_of_the_paper": "In this work, the authors propose a new federated learning algorithm by adopting a neural scheduling technique. In particular, the neural scheduler is trained without needing to access the local data. Preliminary experiments demonstrate the effectiveness of the proposed algorithm.", "summary_of_the_review": "see the comment above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "8HESsJY89t-", "reviewer": "Reviewer_sDWS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- FL has been one of the most important topics in recent years. The motivation for this study to deal with data in hospitals is very understandable and compelling.\n- Making FL frameworks robust against backdoor attacks is particularly an important direction. Learning a scheduler to observe clients and avoid potentially compromised ones could be a good idea (though I'm not yet perfectly convinced).\n- Experimental results on new and real hospital data are invaluable.\n\n### Weaknesses\n\nWhile I found the proposed approach interesting, it is hard to properly assess the significance of the work as many technical points are unclear in the current manuscript. Specific comments are as follows:\n\n#### (1) Teacher\nI don't think that the following argument in the current manuscript is fully supposed: \"The teacher can either be pre-trained or jointly trained with the scheduler. The teacher can also either be trained on the dataset of one node and distributed to the rest or independently trained at each node.\"\n\nSpecifically, the teacher policy specifies batch(es) of data to use as actions (Section 3.3). Then, it should be trained for each client independently since the number of data samples is not necessarily the same across clients. In other words, I cannot see when it is possible to pre-train teachers or share them among clients. Even after data are sorted based on some metrics, each data batch is essentially different among clients, or between pre-training and training data. This makes it questionable if a teacher policy trained with one dataset could be used as-is for another one.\n\nMoreover, because the actions specify the indices of data batches to use, pre-training data and training data should have the same number of batches. Otherwise, there will be some teacher actions whose data batches do not exist (when the number of pre-training batches is larger than that of training batches) or there will be some training batches that won't be selected (when the number of pre-training batches is smaller than that of training batches). I would like to see how the proposed approach deals with these problems.\n\n#### (2) Scheduler\nWhile I think it interesting to learn a scheduler to select nodes, it is not clear how the scheduler is actually used to do so. As described in Section 3.5, the scheduler is a neural network with soft-max outputs with the dimension corresponding to the number of nodes. Does this mean that the scheduler selects only one node per iteration? Or how is it possible to specify multiple nodes? It also remains unclear if this approach could be scalable for hundreds of clients. Overall, the current manuscript lacks such implementation details, which should be addressed in the revised manuscript. \n\n#### (3) Experiments\nCurrently, many technical details are missing in both the main paper and supplementary material. For MNIST and CIFAR-10, how was the distribution of data (or class frequency) different across nodes? Is there any data non-iidness? What specific data were used to pre-train teacher models and autoencoders? The number of clients and data non-iidness are both critical factors for federated learning. Experimental results won't be very strong if they are just about a few clients with iid data.\n\nMoreover, I don't think that current experimental results fully demonstrate the effectiveness of the proposed approach. Since the performances of FLST and RLST (baseline with the student-teacher setup without scheduling) are almost the same, I doubt if the scheduler is working effectively. How can we confirm the effectiveness of the scheduler? Also, it’s not obvious from the results how much the teacher model contributed to the performance. I would like to see an ablation study for how the accuracy changes when only student models were trained without batch selection by the teacher.\n\n\nFinally, it is currently hard to judge the validity of robustness experiments due to the lack of their setup detail. How were model poisoning and data poisoning attacks implemented? What made it difficult (or is it possible) to compare the proposed approach against the existing work on robust FL (Liu et al., 2018; Wang et al., 2019; and Liu et al., 2017b, which are just introduced briefly in the related work section)?", "summary_of_the_paper": "This paper presents a new federated learning (FL) method augmented with \"teacher\" and \"scheduler\". The teacher is a reinforcement learning agent that observes the status of FL clients to inform which data mini-batches the clients should use to train their local model. The scheduler, on the other hand, is a meta-learner observing all the local model updates to decide which nodes (i.e., clients) to distribute the global model. Doing so by the scheduler allows the proposed method to cope with compromised clients. Experimental results using multiple hospital datasets as well as standard image datasets (MNIST and CIFAR-10) show the effectiveness of the proposed approach over some existing methods.", "summary_of_the_review": "My initial score is reject mainly due to the lack of clarity and unconvincing experimental results. As pointed out in the weakness section, the assumptions, implementation details, effectiveness, and limitations of the teacher and the scheduler are not fully presented in the current manuscript. Although experimental results seem to show the effectiveness of the proposed approach for model/data poisoning attacks, I'm not fully convinced by the results due to the lack of their detail and the absence of existing robust FL methods. That said I would like to update my score once all these points are resolved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4KC7uFjqclC", "reviewer": "Reviewer_cKiw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. It is worthwhile to study the scenario of training models across different hospitals based on federated learning so that the privacy of patients is well protected.\n2. Sufficient experiments are conducted to illustrate the performance of the framework as well as the effectiveness of each component.\n3. Most parts of this paper are written in a logical and clear way.\n\nCons:\n1. There is not enough theoretical contribution to either the federated learning field or the curriculum learning field. Most equations in Section 3 were proposed in previous works. \n2. The authors need to point out the original novelty of this paper apart from the existing conclusions they used in the paper. Also, they need to explain how it benefits the community.\n3. The experimental details are not well presented in Sections 4 and 5. For example, the hyperparameters for model training are not given in both the main text and the supplementary material. \n4. The organization of this paper should be improved. For example, the details of datasets should be brief, or at least most of it should be moved to the supplementary material. Also, the discussion and conclusion should also be brief enough to summarize the most important contribution of this paper rather than explaining things in detail.\n\n\nComments:\n1. The teacher component seems like a pre-trained reinforcement learning agent which is fixed during the training procedure. However, when a different student model is selected, it should correspondingly change because there will be a different input state space. This problem should be solved or further explained by the authors.\n3. There are some typos: 1) In Para 4 of Section 1, “Firstly, We use“ -> ”Firstly, we use”; 2) In Para 1 of Section 4, “clinical staff who greet” -> “clinical staffs who greet”; 3) In Para 3 of Section 4, it should not be “samples” in “into three samples of”.\n3. Table 1 shows the performance of decentralized FLST compared with other state-of-the-art centralized classification methods, while Table 2 shows the performance of FLST compared with other state-of-the-art decentralized federated learning methods. It can be seen that FedAvg achieves higher accuracy than most centralised methods, which is different from the results in other works. It is better for the authors to give a further explanation on this.\n", "summary_of_the_paper": "This paper proposes a novel federated learning framework for model training across multiple hospital data centers. There are mainly three components in the framework including a student machine learning model, a teacher reinforcement learning agent, and a scheduler algorithm that directs the teacher to specific data centers. The setup and the details of the whole framework are well presented in the paper. Corresponding experiments are carried out to show the performance of the proposed method.", "summary_of_the_review": "Although It is worthwhile to study the scenario of training models across different hospitals based on federated learning, there is not enough theoretical contribution to either the federated learning field or the curriculum learning field and the organization of this paper should be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work describes an interesting approach of using a reinforcement learning algorithm for federated learning. The paper is well organized and the use-case of performing federated learning while preserving patient privacy is also important. However, the paper has room for improvement. Important baselines used for client selection are missing and so the deep reinforcement learning approach is not well-motivated. Many important technical details are missing such as hyperparameters and distributions for MNIST and CIFAR. The approach is also lacking novelty, DRL has been used for neural scheduling before and the authors do not suggest improvements to that. Finally, the experiments showing robustness to backdoor attacks is unconvincing and can benefit from more analysis.
Decision: Reject