Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel federated learning framework for model training across multiple hospital data centers. There are mainly three components in the framework including a student machine learning model, a teacher reinforcement learning agent, and a scheduler algorithm that directs the teacher to specific data centers. The setup and the details of the whole framework are well presented in the paper. Corresponding experiments are carried out to show the performance of the proposed method.
main_review: Pros:
1. It is worthwhile to study the scenario of training models across different hospitals based on federated learning so that the privacy of patients is well protected.
2. Sufficient experiments are conducted to illustrate the performance of the framework as well as the effectiveness of each component.
3. Most parts of this paper are written in a logical and clear way.

Cons:
1. There is not enough theoretical contribution to either the federated learning field or the curriculum learning field. Most equations in Section 3 were proposed in previous works. 
2. The authors need to point out the original novelty of this paper apart from the existing conclusions they used in the paper. Also, they need to explain how it benefits the community.
3. The experimental details are not well presented in Sections 4 and 5. For example, the hyperparameters for model training are not given in both the main text and the supplementary material. 
4. The organization of this paper should be improved. For example, the details of datasets should be brief, or at least most of it should be moved to the supplementary material. Also, the discussion and conclusion should also be brief enough to summarize the most important contribution of this paper rather than explaining things in detail.


Comments:
1. The teacher component seems like a pre-trained reinforcement learning agent which is fixed during the training procedure. However, when a different student model is selected, it should correspondingly change because there will be a different input state space. This problem should be solved or further explained by the authors.
3. There are some typos: 1) In Para 4 of Section 1, “Firstly, We use“ -> ”Firstly, we use”; 2) In Para 1 of Section 4, “clinical staff who greet” -> “clinical staffs who greet”; 3) In Para 3 of Section 4, it should not be “samples” in “into three samples of”.
3. Table 1 shows the performance of decentralized FLST compared with other state-of-the-art centralized classification methods, while Table 2 shows the performance of FLST compared with other state-of-the-art decentralized federated learning methods. It can be seen that FedAvg achieves higher accuracy than most centralised methods, which is different from the results in other works. It is better for the authors to give a further explanation on this.

summary_of_the_review: Although It is worthwhile to study the scenario of training models across different hospitals based on federated learning, there is not enough theoretical contribution to either the federated learning field or the curriculum learning field and the organization of this paper should be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose a new federated learning algorithm by adopting a neural scheduling technique. In particular, the neural scheduler is trained without needing to access the local data. Preliminary experiments demonstrate the effectiveness of the proposed algorithm.
main_review: This work applies federated learning with a neural scheduler to improve the effectiveness and robustness of the global model.  The idea is interesting, but they're still several concerns based on the current submission:
1. Adopt deep reinforcement learning (DRL) as a neural scheduler is not new.  In [1], the authors also adopt DRL technique to filter the data or gradient boost the training speed and performance. It seems that the main contribution of this work is to extend the DRL-based neural scheduler to the federated learning setting, which limits the novelty of the proposed work. Hence, we recommend the authors give more related works and detailed comparisons on this submission with existed (DRL-based) neural scheduler. 
2. The experiments are not sufficient to demonstrate the efficacy of the proposed algorithm. In fact, there exist a large amount of work that uses hypernetwork [2] or bandit [3,4,5, etc] techniques to perform client selection.  Hence, we recommend the authors give baselines to demonstrate the efficacy of the proposed algorithm.
3. The experients demonstrate the proposed algorithm is robust to backdoor attacks. This phenomenon is interesting. However, the reason why this approach works is still unknown. We recommend the author gives more explanation on these observations either in theory or in practice. 

[1]  Fan Y, Tian F, Qin T, Bian J, Liu TY. Learning what data to learn. arXiv preprint arXiv:1702.08635. 2017 Feb 28.
[2] Shamsian, A., Navon, A., Fetaya, E. and Chechik, G., 2021. Personalized Federated Learning using Hypernetworks. arXiv preprint arXiv:2103.04628.
[3] Xia W, Quek TQ, Guo K, Wen W, Yang HH, Zhu H. Multi-armed bandit-based client scheduling for federated learning. IEEE Transactions on Wireless Communications. 2020 Jul 16;19(11):7108-23.
[4] Huang T, Lin W, Shen L, Li K, Zomaya AY. Stochastic Client Selection for Federated Learning with Volatile Clients. arXiv preprint arXiv:2011.08756. 2020 Nov 17.
[5] Yoshida, N., Nishio, T., Morikura, M. and Yamamoto, K., 2020, December. MAB-based Client Selection for Federated Learning with Uncertain Resources in Mobile Networks. In 2020 IEEE Globecom Workshops (GC Wkshps (pp. 1-6). IEEE.

summary_of_the_review: see the comment above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a new federated learning (FL) method augmented with "teacher" and "scheduler". The teacher is a reinforcement learning agent that observes the status of FL clients to inform which data mini-batches the clients should use to train their local model. The scheduler, on the other hand, is a meta-learner observing all the local model updates to decide which nodes (i.e., clients) to distribute the global model. Doing so by the scheduler allows the proposed method to cope with compromised clients. Experimental results using multiple hospital datasets as well as standard image datasets (MNIST and CIFAR-10) show the effectiveness of the proposed approach over some existing methods.
main_review: ### Strengths

- FL has been one of the most important topics in recent years. The motivation for this study to deal with data in hospitals is very understandable and compelling.
- Making FL frameworks robust against backdoor attacks is particularly an important direction. Learning a scheduler to observe clients and avoid potentially compromised ones could be a good idea (though I'm not yet perfectly convinced).
- Experimental results on new and real hospital data are invaluable.

### Weaknesses

While I found the proposed approach interesting, it is hard to properly assess the significance of the work as many technical points are unclear in the current manuscript. Specific comments are as follows:

#### (1) Teacher
I don't think that the following argument in the current manuscript is fully supposed: "The teacher can either be pre-trained or jointly trained with the scheduler. The teacher can also either be trained on the dataset of one node and distributed to the rest or independently trained at each node."

Specifically, the teacher policy specifies batch(es) of data to use as actions (Section 3.3). Then, it should be trained for each client independently since the number of data samples is not necessarily the same across clients. In other words, I cannot see when it is possible to pre-train teachers or share them among clients. Even after data are sorted based on some metrics, each data batch is essentially different among clients, or between pre-training and training data. This makes it questionable if a teacher policy trained with one dataset could be used as-is for another one.

Moreover, because the actions specify the indices of data batches to use, pre-training data and training data should have the same number of batches. Otherwise, there will be some teacher actions whose data batches do not exist (when the number of pre-training batches is larger than that of training batches) or there will be some training batches that won't be selected (when the number of pre-training batches is smaller than that of training batches). I would like to see how the proposed approach deals with these problems.

#### (2) Scheduler
While I think it interesting to learn a scheduler to select nodes, it is not clear how the scheduler is actually used to do so. As described in Section 3.5, the scheduler is a neural network with soft-max outputs with the dimension corresponding to the number of nodes. Does this mean that the scheduler selects only one node per iteration? Or how is it possible to specify multiple nodes? It also remains unclear if this approach could be scalable for hundreds of clients. Overall, the current manuscript lacks such implementation details, which should be addressed in the revised manuscript. 

#### (3) Experiments
Currently, many technical details are missing in both the main paper and supplementary material. For MNIST and CIFAR-10, how was the distribution of data (or class frequency) different across nodes? Is there any data non-iidness? What specific data were used to pre-train teacher models and autoencoders? The number of clients and data non-iidness are both critical factors for federated learning. Experimental results won't be very strong if they are just about a few clients with iid data.

Moreover, I don't think that current experimental results fully demonstrate the effectiveness of the proposed approach. Since the performances of FLST and RLST (baseline with the student-teacher setup without scheduling) are almost the same, I doubt if the scheduler is working effectively. How can we confirm the effectiveness of the scheduler? Also, it’s not obvious from the results how much the teacher model contributed to the performance. I would like to see an ablation study for how the accuracy changes when only student models were trained without batch selection by the teacher.


Finally, it is currently hard to judge the validity of robustness experiments due to the lack of their setup detail. How were model poisoning and data poisoning attacks implemented? What made it difficult (or is it possible) to compare the proposed approach against the existing work on robust FL (Liu et al., 2018; Wang et al., 2019; and Liu et al., 2017b, which are just introduced briefly in the related work section)?
summary_of_the_review: My initial score is reject mainly due to the lack of clarity and unconvincing experimental results. As pointed out in the weakness section, the assumptions, implementation details, effectiveness, and limitations of the teacher and the scheduler are not fully presented in the current manuscript. Although experimental results seem to show the effectiveness of the proposed approach for model/data poisoning attacks, I'm not fully convinced by the results due to the lack of their detail and the absence of existing robust FL methods. That said I would like to update my score once all these points are resolved.
review_title: null

Meta-review: This work describes an interesting approach of using a reinforcement learning algorithm for federated learning. The paper is well organized and the use-case of performing federated learning while preserving patient privacy is also important. However, the paper has room for improvement. Important baselines used for client selection are missing and so the deep reinforcement learning approach is not well-motivated. Many important technical details are missing such as hyperparameters and distributions for MNIST and CIFAR. The approach is also lacking novelty, DRL has been used for neural scheduling before and the authors do not suggest improvements to that. Finally, the experiments showing robustness to backdoor attacks is unconvincing and can benefit from more analysis.
Decision: Reject