Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tries to address the big model on the server-side in federated learning (FL) via knowledge distillation. Selective transfer knowledge happens on both server and clients sides. The proposed model maintains a set of global, correct logits so that it can be used as memories for the server's distillation. When the server predicts wrong on the public datasets, it transfers the knowledge from the correct, weighted clients. The proposed model is evaluated on image datasets with many FL baselines.
main_review: ++ Many experiments are conducted to show the results on performance, robustness against poisoning attacks, comm. cost, the model behavior in different server sizes.

However, major concerns are existing in the current form of this paper:

-- The contribution is not clear. The key issues of existing methods and the challenges of the proposed method are not clearly stated.

-- The technical depth seems weak. Table 1 is a good summary, but it also shows the weakness of the proposed model.

Smaller ones:

The mathematical expressions need to be improved, which hinders the understanding of the content.

The notations are not consistent. For example, f_s and f^0 are both used to denote the server model.

No explanations for symbols L_DL, \epsilon, et al.

The y_i in Section 3.1 should be y_i^k

summary_of_the_review: contribution not clear, technical depth weak, writing and notation to be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a KD-based FL framework for training a larger server model securely. The proposed framework is a combination of FedGKT-like idea and further apply some KD strategies for selective transfer. Many baselines are compared to demonstrate its efficacy.
main_review: * Strengths
1. The selective KD is novel to FL
2. The problem (training a large-scale model) by utilizing scatter data from large number of edge devices is well motivated.
3. Many important and highly related baselines are compared.

* Weaknesses
1. It's supervising to me that a smaller model (ResNet-20) used in the proposed framework can outperform a larger model (ResNet-56) used in  FedDF and FedGKT.

2. FedGKT's accuracy is not the same as the result reported in the original paper. Is that because of bug or not well-tuned?

3. The proposed framework is not data-free KD like FedGKT. The public and private data is 1:1. In practice, this may not be the case. Another issue of such setting is that the performance in other ratio is not clear. It would be better to do an ablation study to understand the performance in different ratio.

4. The communication cost is over claimed. FedGEMS requires a public dataset for each client to boost the transfer, when the number of clients is big, the cost is very high. The total communication cost should be larger than FedGKT. The benefit of FedGKT is that it only replies on the sample number on each client, which is normally small. Another concern is the memory cost. FedGKT does not need any public data to do KD, however FedGEMS requires too many samples for edge devices (30000 in the current implementation).

5. Another issue would be the scalability. Note that cross-device FL should be stateless, meaning that the newly sampled clients do not have any previous cached states (optimizer states, logiits, etc) [1] . FedGEMS's assumption seems can only be used in cross-silo FL.

6. It seems the author uses FedML to finish the experiments. Does FedML support the implementation? Especially the complex communication and KD pattern described in FedGEMS.

7. It's highly appreciated when the authors can discuss some limitations. For example, the memory cost for public data; whether the client side can support a large model; what if the public and private data ratio is changed, etc.

8. The Introduction attempt to convince readers that training GPT-3 like giant model in FL is necessary. This is ok to me. But the experiments only talk about small scale ResNet-20. Note that GPT-3 is a billion-level model, while ResNet-20 is much smaller. This somehow contradicts the motivation. Another related issue is that the paper is titled as "larger" model but the model used in experiments (ResNet-20) is smaller than other baselines.

9. The writing need to be further polished.

10. "Continual" in Abstract may not be a correct word, since it normally refers to continual ML. But FedGEMS does not have related design.

[1] Federated Reconstruction: Partially Local Federated Learning. NeurIPS 2021.

summary_of_the_review: Overall, it's a good empirical paper, but it's better to address some of the above concerns before applying the proposed framework in practice.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an empirical technique to train large server models with  selective knowledge fusion. The paper passes logits predicted on a public dataset accessible to both clients and server to share knowledge and uses a selection scheme to withstand poisoning attacks.
main_review: Strengths:
- The authors propose an empirical novel selection technique for knowledge fusion.
- The authors tackle multiple challenges in federated learning with their approach.Their results seem to show improved accuracy for both hetero and homogeneous settings. Since only predicted logits are shared, the communication cost is also low.
- The algorithm is also shown to work well in resisting poisoning attacks.

Weakness:
- As per my understanding, the algorithm at the server end uses a combination of techniques. It could be very useful for future researchers to see the impact of each of them separately. For example, we can analyze the effects on accuracy if the logits are not stored (Algorithm 1, Lines 8-9) or the client selection step is not included and all clients are used instead (Algorithm 1, Lines 11-12) 
- It looks like the server model is only trained on the public data. What will happen if the server model overfits and predicts everything or most of the samples in the public data correctly? As per my understanding, it will not use the client's information from the logits at all after that. 
- The selective knowledge fusion technique is very empirical. The paper therefore needs to have many more ablation studies. For example, the paper could show the effect of larger and smaller server models and the change in accuracy for different numbers of clients during poisoning attacks.
- In Page 5, Paragraph 1, Line 2-3, it is mentioned that the unreliable clients are removed. I understand the reasoning is to remove bad clients but it seems we may be losing some information here. For complex datasets, we may have a scenario where, for outlier samples, generalized clients’ models are not predicting correctly but overfitted clients’ models are predicting better. In that case the server model might not be able to generalize well.
- The title, although technically okay, gives the impression that the server model is very large. But the server model(Resnet-20) is not exactly a whole lot larger than the largest client models (Resnet-11 to Resnet-17). For reference FedGKT uses ResNet-56. As mentioned before, there needs to be more thorough ablation studies for larger and smaller server models.
- In Table 2, the results for FedGKT look to be different from the ones mentioned in the actual FedGKT paper. The exact changes to the training and inference process is not mentioned.


summary_of_the_review: This paper proposes a novel approach to tackle several problems in federated learning using selective knowledge fusion. However, the algorithm proposed is largely empirical and needs more thorough ablation studies to strengthen its core claims.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to enable training of higher accuracy models by training a high-capacity model at the server and lower capacity models at the clients. Knowledge is shared via distillation on a weighted average of logits provided by the clients, rather than by averaging model parameters or model differences. The approach assumes the existence of a labeled public dataset. Experiments illustrate the promise of this approach.
main_review: The problem considered in this paper is interesting and important. The need to accommodate on-device systems constraints (limited memory and computational power) significantly limits the performance of models trained using FL. This paper suggests an interesting approach towards overcoming this issue, and the experimental results suggest that the proposed approach is promising.

However the paper has a number of limitations, which are detailed below. Some are related to the practicality of the proposed approach, but the majority of these are limitations of the experimental setup used for comparisons. Normally I would be willing to overlook some of the concerns, but there are so many in this case that my overall recommendation is to reject the paper.

**Major concerns:**
1. The scheme described in Sec 3.3.2 does not seem compatible with privacy considerations that are important in FL. In particular, in order to partition clients into reliable and unreliable sets as described, the server would need to have the individual logits for each training instance from each client. Such information could be pretty easily used to reverse-engineer the clients' training data (e.g., Geiping et al., NeurIPS 2020).

2. If I understand correctly, in all experiments, the distribution of the public data set is identical to that of the private dataset (if the private datasets were all pooled together). This seems very realistic, as one might expect that clients could have data following some distribution locally which is not reflected at all by data in the public dataset; in particular, I'm thinking of clients with private training samples with inputs $x_i$ that fall outside of the distribution of inputs in the public dataset.

3. The experiments use a very small number of clients (only 8), and each has a relatively large amount of data. No information is provided in the main paper about how much training effort is required to achieve the results in Table 2. (How many global rounds, and how many local epochs per round?) Also, the introduction and setup considered seem to imply that this approach is intended for cross-device FL. If that is truly the case, it would be important to include experiments showing that FedGEMS works with many more clients and less data per client. The current results seem to suggest that it may be useful for cross-silo FL.

4. The performance reported for FedGKT is _much_ lower than what is reported in [He et al., 2020](https://arxiv.org/abs/2007.14513). I realize the models may not be exactly the same, but I wouldn't expect to see such a marked difference. This substantial difference, without any remark from the authors, leads me to question how much faith to put in the experimental results. How much effort went into hyperparameter tuning for each method? 

5. Are the attacks considered in Sec 4.3 being newly introduced in this paper or have they been considered in previous literature? If they are, references should be included in the main text of the paper (not just in the appendix). If the attacks are actually new, then significantly more motivation should be included for why these attacks are relevant.

6. Table 3 reports the results of poisoning attacks in the homogeneous setting, but it would be much more interesting and relevant to also include the performance in the heterogeneous setting.

7. Fig 3 shows the communication cost per epoch (per round?). This is only part of the story, though, since the overall communication overhead also depends on the number of rounds required to reach the final accuracy. The paper does not report how many rounds each method was run for, or if it's the same for all methods. Upon further examination in Table 8 of the Appendix, it appears that different methods were run for different numbers of rounds. It is not clear that this leads to a fair comparison.

8. Fig 4 shows that the number of samples selected for clients to report back to the server changes during training. Fig 3 shows a fixed communication cost per iteration. Are those numbers at the beginning or end of training, or averaged over an entire training run? Also, wouldn't one expect those values to change substantially depending on the problem and the public/private datasets?

9. The objectives (2), (5), and (6) all depend on the parameter $\epsilon$ that balances between cross-entropy and distillation losses. What value was used in the experiments? It is impossible to reproduce the results without knowing this. Also, how sensitive is performance to the choice of this parameter? 

**Minor / requests for additional information:**
* This paper seems to implicitly focus on classification problems. This ought to be stated explicitly in the introduction, and ideally also in the abstract.
* The title ("Client-side...") of Sec 3.3.2 is confusing, since that section describes operations that are all performed at the server.
* Please add an x-label for Fig 3.
* Do the communication amounts for FedGEMS reported in Fig 3 count the overhead of the server sending the selected indices to each client?
summary_of_the_review: The problem considered in this paper is interesting and important. The need to accommodate on-device systems constraints (limited memory and computational power) significantly limits the performance of models trained using FL. This paper suggests an interesting approach towards overcoming this issue, and the experimental results suggest that the proposed approach is promising.

However the paper has a number of limitations. Some are related to the practicality of the proposed approach, but the majority of these are limitations of the experimental setup used for comparisons. Normally I would be willing to overlook some of the concerns, but there are so many in this case that my overall recommendation is to reject the paper.
review_title: null

Meta-review: This paper proposes a knowledge distillation strategy to enable the use of a large server-side model in federated learning while satisfying the computation constraints of resource-limited clients. The problem is relevant and well-motivated, and the paper presents compelling experimental results to support the proposed strategy. However, reviewers had the following major comments suggestions/:
1) The theoretical analysis section needs improvement in terms of the technical depth and rigor
2) Better explanation of how the proposed strategy compares with previous works/baselines
3) Considering the privacy and scalability properties of the proposed strategy.

The paper generated lots of constructive post-rebuttal discussions between the authors and the reviewers, and I believe the authors received several ideas to improve the work and appreciated the reviews. One of the reviewers increased their score. However, based on the current scores, I still recommend rejection. I do think the paper has promise, and with improvements, the revised version will make an excellent contribution.
Decision: Reject