Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a generalized federated distillation framework which divides local nodes into groups: the aggregation is first inside each group and then from groups into the final global model.  The advantage is groups can do each aggregation in parallel in the first stage.
The experiments exhaust several popular federated distillation method for the aggregation. The results show the proposed group-division ensemble strategy does improve over the ensemble directly from all locals.   
strength_and_weaknesses: Strength: 
1. The proposed group-division ensemble framework can be generalized to different existing FL methods. First, aggregation inside individual group helps to improve the level of parallel.  
2. Experiments show the proposed methods achieves better performance than directly aggregating all locals.

Weakness:
1. Limited novelty: the authors only replace the typical aggregation in FL with two stages: group aggregation and then global aggregation. The detailed ensemble strategies (e.g. bayesian ensemble) are borrowed from existing works. The novelty and contribution is limited considering the method is just simple combination.  
2. Overclaim about scalability: I cannot see any improvement on scalability from the proposed method and experiments design.
3. Overclaim about privacy: In abstract it says "FedEED also achieves higher level of privacy protection, because the access to client models is no longer required." But the method still iteratively upload local model parameters besides the group global model. [1][2] demonstrated that local private data could be fully recovered from publicly-shared gradients (iteratively shared local model parameters). 
3. Overclaim about efficiency: No quantitative analysis of efficiency metrics such as latency, communication bandwidth. Considering the "ensemble trainer" also involves the participation of local models, I doubt if the latency and communication bandwidth could be even higher than the existing FL methods for comparison. It is meaningless to show #rounds in the table, as the latency and bandwidth of each round in your method is obviously higher than other FL methods. The communication cost in Table3 is very ambiguous without any analysis or calculation.
4. Experiments: only have experiments on cifar10/cifar100. Should do more through experiments under typical FL setting: such as EMNIST in SCAFFOLD or NLP tasks (AG News, SST2 in FedDF). 

[1] Deep leakage from gradients.
[2] Inverting gradients–how easy is it to break privacy in federated learning?
clarity,_quality,_novelty_and_reproducibility: Quality: generally good.
Clarity: clear written except some claims about scalability, privacy, efficiency. (see weakness)
Originality: limited novelty, even the group-division ensemble is not new (Fed-ensemble proposes similar group division method). Could you explain the difference with Fed-ensemble?
summary_of_the_review: Although the experiments show better performance over the baseline methods, the novelty is very limited. And the efficiency metrics are not clear. I also doubt the claims regarding privacy and scalability.   Since the "efficiency" is the main point of the paper, I hope the authors could provide more efficiency analysis with quantitative results on latency, communication bandwidth.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors studied the key components of distillation-based model aggregation in FL. Different from existing approaches, the ensemble teacher of FedEED is constructed by aggregated models, instead of client models, to achieve improved scalability in large-scale systems.
strength_and_weaknesses: Strength:
Due to the use of aggregated models, FedEED also achieves a higher level of privacy protection. In addition, the knowledge distillation in FedEED only happens from the ensemble teacher to a designated model such that the diversity among different aggregated models is maintained.

Weaknesses:
The authors evaluated the performance of the proposed method only on the task of Non-IID labels and did not validate the performance of the proposed method on the Non-IID feature task, which is a more realistic scenario because there is often a large domain gap problem in the distribution of data features of different clients as the different data acquisition devices, environments, and qualities. Therefore it does not provide a comprehensive assessment of the generality of the proposed method.
clarity,_quality,_novelty_and_reproducibility: The problem this paper address is clear. The logic of the article is clear. Somewhat novelty. Repeatable in a limited simulation scenario.
summary_of_the_review: 1.  The authors should add experiments to further validate the performance of the proposed method on Non-IID feature tasks to further evaluate the generality of the proposed method.
2. The authors' presentation is not clear and the architecture of the algorithm cannot be well understood. Perhaps the authors can give a clear architecture diagram to assist the readers to understand the essence of the proposed method clearly.
3. Several general FL methods have been proposed and achieved excellent performance, such as FedBN, FedRep, and FedDC, and the authors should analyze the performance of more SOTA FL methods to further evaluate the proposed approach.

a) Li X, Jiang M, Zhang X, et al. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.
b) Collins L, Hassani H, Mokhtari A, et al. Exploiting shared representations for personalized federated learning. International Conference on Machine Learning. PMLR, 2021: 2089-2099.
c) Gao L, Fu H, Li L, et al. FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10112-10121.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper first propose a generalized distillation framework. Based on the study of each component in the generalized framework, this paper proposes Federated Efficient Ensemble Distillation(FedEED), which is efficient, scalable and privacy.
strength_and_weaknesses: Strengths:

1. This paper attempts to achieve generalizability of federated distillation, which is a very innovative study.

2. The application of knowledge distillation in the field of federated learning is well researched and summarized.

3. A number of experiments are done using several federated distillation algorithms.

Weaknesses:

1. In abstract
	- This paper first proposes a generalized distillation framework which divides the federated distillation process into three key stages. However, there are various types of federated distillation, and the three-stage framework is only one of them, so it should not be called a generalized distillation framework. 
	- FedEED also achieves a higher level of privacy protection, because the access to client models is no longer required. However, The client model still needs to be transferred between local trainer and  ensemble trainer, which means the client model can still be attacked.

2. In framework
	- This paper describes in detail the three-stage process of the generalized federated distillation framework, wouldn't it be better to add a framework figure to illustrate it more visually? 
	- This framework includes K local trainers, an ensemble trainer, and a global trainer, but traditional federated learning includes K clients and a server, what is the correspondence between them, this paper does not explain.

3. In experiments
	- This paper mentions that the number of clients is 20, with 40% of activated clients, which means that there are very few clients, which is contrary to this paper's emphasis on the fact that FedEED achieves improved scalability in large-scale systems. Therefore, experiments on more clients should be conducted to demonstrate the scalability of FedEED.
	- The correspondence between Figure 1 and Table 1 is not clear, and the introduction part is also a bit messy, which is easy to cause confusion. If you follow the order in Figure 1, the order in Table 1 would be “Global model(K=1)、Ensemble(K=1,Clients)、Ensemble(K=1,Clients,Weighted)、Ensemble(K=1,Bayesian,Gaussian)、Ensemble(K=1,Bayesian,Dirichlet)、Global model(K=4)、Ensemble(K=4,Clients)、Ensemble(K=4,Aggregated)、Ensemble(K=4,R=2,Aggregated)、Ensemble(K=4,R=4,Aggregated)”.
	- The dataset CIFAR100 is included in the description of Table 2, but it is not available in the experimental results. In addition, only Table 4 conducted experiments on CIFAR100, why not include this dataset in other experiments.
clarity,_quality,_novelty_and_reproducibility: Limited clarity, Limited quality, Limited novelty, Highly reproducibility.
summary_of_the_review: This paper presents an innovative research of ederated distillation, But I have doubts about the scalability and privacy of the proposed framework FedEED.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates applying knowledge distillation to federated learning. It summarizes many related works to give a set of common building blocks and experiments with different choices for each block on ResNet20/56 + Cifar10/100 to see their impact on the accuracy.
strength_and_weaknesses: Strength:
- There is a good effort to summarize the common building blocks across many knowledge-distillation-based FL algorithms. This is helpful for readers to get a high-level overview of algorithms in this class.
- This paper also provides a few popular methods that can be used for each block and performs experiments to compare these methods. The results may be useful when readers try to adapt an existing method to their applications or systems.

Weakness:
- The main issue of this paper is that it is difficult to read and understand. The paper is flooded with undefined terms, symbols, and confusing statements. Also, there is no figure to serve illustration. For example, the paragraph in Sec. 3 states that "In every training round, each of the local trainers will distribute one of the global models to a subset of clients". For FL, the server is typically responsible for distributing models. Does this sentence mean one client will choose a model from the global models and distribute it to other clients, involving something like client-to-client communication? Why does a client store global models in addition to their local models? Another example is that the terms like "global model", "ensemble model", "local trainer", "ensemble trainer", and "global trainer" are not defined, and the symbols in the pseudo code are not defined either.
- There are many unsupported or unexplained claims. For example, Sec. 4.1 has the sentence "Existing distillation-based aggregation methods require all client models to build the ensemble, causing privacy concerns.". Why does this approach cause privacy concerns and why doesn't the proposed approach are not explained nor supported by references. Another example is also from Sec. 4.1. "We noticed the idea of Fed-ensemble, which builds the ensemble from multiple global models for inference purposes. This solves the privacy and scalability issues.". Again, there is no explanation about why this solves the privacy and scalability issues.
- Most of the experiments are on the small CIFAR10 dataset plus the small ResNet20 model except for Table 4, which contains ResNet56 and Cifar100. Because the authors never mention what the metric for the accuracy is, I assume it is top-5 accuracy. ResNet56 on CIFAR100 can easily get a top-5 accuracy above 90%, but the numbers in Table 4 are at the level of 60%. Therefore, I am skeptical about whether the conclusions drawn from these experiments can be generalized to practical settings.
- In Sec. 6.2, there is only one sentence "Ablation study will be provided in the appendix.". The ablation study is an important part of the main paper. Personally, I don't think moving it to the appendix to save space is a good practice. I leave it to the chair to make the judgment.
clarity,_quality,_novelty_and_reproducibility: Clarity and quality:
- As stated in the weakness section, I don't think this paper is clear and easy to understand.

================================================================

Novelty:
- The technical novelty is limited. Although this paper summarizes existing works and experiments with several existing methods, it does not present any novel methods or novel ways of applying existing methods.

================================================================

Reproducibility:
Yes if the authors can improve the wording.
summary_of_the_review: Although it is a good effort to summarize related works and experiment with multiple design choices, the technical novelty is somewhat limited. Moreover, this paper needs to be revised to make it easier to understand and better explain and support the claims.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new federated learning technique that does the aggregation of client models via knowledge distillation in stages. 

Strength:
* Nice survey of knowledge distillation techniques for federated learning
* Extensive experiments with many knowledge distillation techniques for federated learning

Weaknesses:
* Lack of novelty (dividing the aggregation into stages is not particularly novel)
* Unsupported claim that the proposed technique offers better privacy than other federated learning techniques
* Unsupported claim of improved efficiency (no timing results)
* Unclear manuscript

Unfortunately, this work does not have enough novelty for publication and it suffers from unsupported claims as well as a lack of clarity.
justification_for_why_not_higher_score: * Lack of novelty (dividing the aggregation into stages is not particularly novel)
* Unsupported claim that the proposed technique offers better privacy than other federated learning techniques
* Unsupported claim of improved efficiency (no timing results)
* Unclear manuscript
justification_for_why_not_lower_score: NA
Decision: Reject