Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper considers the problem of training a global model with only one-shot communication between the server and clients in the federated learning setting. This problem is addressed by designing a generator based on the ensemble of 'trained' local models, to generate synthetic data samples, which can then be further utilized to distill the ensemble models to the global model in a knowledge distillation paradigm.
strengths_and_weaknesses: [Strengths]

- The one-shot federated learning is an important problem in both areas of machine learning and distributed computing.

- Using a generator trained from local models can provide synthetic data samples for the knowledge distillation between the ensemble models and the global model, which is also a simple solution for data-free distillation.

- The evaluation has been conducted on various datasets, which is good.

[Weaknesses]

- The novelty is limited. The setting of 'data-free distillation' + 'heterogeneous local models' has been already addressed by some previous works. For example, FEDGEN [1] also trains a generator to provide data-free knowledge distillation between the server and clients, wherein, their local models are also in heterogeneous architectures. The idea is almost the same.

- The one-shot relies heavily on the 'well-trained' local models, to reduce the communication overhead between the server and clients. However, such a setting is not so practical when the local devices are resource constrained, in either data or computing capability.

- Using the ensemble output results of uploaded local models may cause serious privacy issues. In that case, each individual local model can be examined on both input and output, thus the attackers can have the potential to reconstruct local models or local samples if the server is adversary-oriented.

- The writing and presentation should be improved.

[1] Data-Free Knowledge Distillation for Heterogeneous Federated Learning, ICML 2021.
questions: 

Please refer to item **'Weaknesses'** in the previous section.
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper focuses on one-shot federated learning, i.e., the server can learn a model with a single communication round. The proposed FedSyn method has two stages: first, training a generator from the ensemble of models from clients; second, distilling the knowledge of the ensemble into a global model with synthetic data. The authors validate the efficacy of FedSyn by conducting extensive experiments on 6 different datasets with various non-IID settings generated from Dirichlet distributions. Results can well support that the proposed method consistently outperforms all the baselines.
strengths_and_weaknesses: Strengths
1. This paper focuses on one-shot FL, an interesting but less explored topic. From my understanding, the proposed method is by far the most practical one considering that：1) DENSE requires no additional information to be transferred between clients and the server; 2) DENSE does not require any auxiliary dataset for training; 3) DENSE considers both model heterogeneity, i.e., different clients with different model architectures. Generally, I think the investigated problem is sound and interesting. I think this can be an extremely strong paper in one-shot FL.

2. The method using data-free ensemble distillation is inspiring and novel. The experimental evaluation of applying the data-free ensemble distillation to one-shot FL significantly improves the performance of the global model. Fig. 1 illustrates the method clearly and is well drawn. 

3. The experimental settings are well motivated, and the related analysis is convincing. My favorite parts of the paper are the discussion and the ablation study, which offer sound support for the proposed DENSE. 

Overall, this paper is very interesting and to my knowledge novel. It seems like a pioneering contribution towards practical one-shot federated learning. Hence, I would like to vote for strong acceptance.  

Weaknesses.

Below questions need to be addressed to further improve the quality of this paper.
1. What is the architecture of GAN? Is a larger GAN better for model distillation?

2. In Fig.3, why the performance of Fedavg decreases with longer training epochs?

3. In Table 6, how about the results for "w/o l_CE"? Is l_CE necessary?

4. Even though authors provide sufficient references, a few important papers are recommended. For example, a recent IJCAI paper “Data-Free Adversarial Knowledge Distillation for Graph Neural Networks” that also studied data-free distillation. These papers did not study exactly the same topic as this paper, but would certainly further enrich the literature review.

questions: 1. What is the architecture of GAN? Is a larger GAN better for model distillation?

2. In Fig.3, why the performance of Fedavg decreases with longer training epochs?

3. In Table 6, how about the results for "w/o l_CE"? Is l_CE necessary?

4. In the future direction, authors mentioned “defend privacy attacks in one-shot FL”, could you elaborate more on this? From my understanding, one-shot FL is already more secure than multi-round FL.  

limitations: The analysis, theory, and method are sound to me, but I didn't check the privacy concern.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a novel data-free one-shot FL framework named DENSE based on data generation and knowledge distillation techniques so that it can be applied to communication-efficient heterogeneous FL. Evaluation results on multiple benchmark datasets illustrate the good performance of the proposed FL framework.
strengths_and_weaknesses: Strengths:
[1] This paper proposed a novel data-free one-shot FL framework consisting of two stages: i) trains a generator based on ensemble models uploaded from clients and random labels; ii) adopts knowledge distillation to transfer the output from the teacher model to the global model. The idea of data-free one-shot FL is very interesting. 
[2] Extensive experiments have been conducted to evaluate the performance of the proposed framework

Weakness:
[1] I am concerned about the privacy as the ensemble models is uploaded to the central sever. As far as I know, attackers in the server side can still recover the privacy data based on the model parameters.
[2] I am wondering if the labelled data in the server would leak the data privacy? Such as race and gender.
[3] Please explain the important steps in algorithm 1 in Appendix with mode details. It seems hard to understand the algorithm.
[4] Please conduct experiments with mutiple random seeds, and then report the mean and variance. Otherwise, the experimental results seem not very convincing. 
questions: Why not compare the proposed method with the FedGen method that is data-free FL with knowledge distillation? I know it requires to send the model parameters of generator between server and clients multiple rounds.
limitations: Yes, the authors have described the limitations of this work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed a novel one-shot data-free Federated Learning algorithm:
1. The algorithm only requires a single communication rounds. The clients locally train the model and uploads the model to the central server.
2. In the server, the ensemble of the client model is utilized to train a generator model that generates synthetic data. The generator is trained by giving a random number and a random label, such that the logits computed from the generated image by averaging the client models minimize the cross entropy.
3. Then the generator is used to generate a large amount of synthetic data to be labeled by the client ensemble. One then distill the knowledge of the ensemble to a global model by train it with synthetic data.
4. The authors also includes additional loss terms for generator training to ensure similarity, stability, transferability of the data.
5. Finally, extensive experiments are carried out, comparing with existing Federated learning algorithm as well as ablation study.
strengths_and_weaknesses: The paper shows strength in the following perspectives:
1. Originality. The algorithm proposed is novel to my best knowledge. It addresses several issues of applying federated learning to real world applications. First, the algorithm is genuinely data free: no client data, distillation data or other data summary will be uploaded to server. Second, the algorithm admits heterogeneity of client models, such that in real world applications clients have the freedom of picking local models. Thirdly, the algorithm is one-shot in the sense that only one round of communication is needed, although the author mentioned that more rounds can improve the quality of the model.
2. Quality. The experiments conducted in the paper are extensive and sound. The author explains why some of the comparisons are not included such as regularization based methods. There are also plenty of experiments with a wide range of parameter space to demonstrate the effectiveness of the algorithm. Those include balancedness of the client data, number of clients, number of rounds and effects of added terms.
3. Clarity. The paper is well-written. The author reviews the existing limitations of the literature, and gives a clear structure on the components of the algorithm.
4. Significance. The paper's result is important both from academic and practical point of view. For the latter, it will stimulate the research community for real world application of FL, such that the whole infrastructure is secure, privacy-preserving, and high quality.

The weakness of the paper, in my opinion, has the following aspects:
1. Quality. The paper does not have a lot of theoretical analysis, especially on why the algorithm would converge regardless of which dataset one is using, and data distributions among clients, etc. 
2. Clarity. While it is OK to move the backgrounds and preliminaries in appendix, in my opinion it would be better to move the full algorithm (Algorithm 1) in the main context - readers may constantly go to the table and refer to it.
questions: There are a few questions regarding the technical details:

1. The generator G(z) is trained by first sample random pairs of random number and label (z, y). Then the generated input x = G(z) is used such that x would minimize some loss. The author first shows cross entropy loss alone does not work well and conjecture that it is due to non-IID data. However, in ablation study one could certainly make the data IID for all clients (table 6). How does table 6 obtain the results? What is the hyperparameter used? 

In my opinion the poor performance of cross entropy alone could also due to the sampling setup: we generate a random pair (z, y). There may not exist a proper transformation between z to x such that (z, y) is independent. In other words, (z, y) could be correlated. 

2. How does the author handle the case that for client models, their logits distributions are very different (line 105-106)? In extreme case, suppose one model produce a set of logits of value O(1e6) while other models produce logits of value (1e-6). Then one model could clearly dominate over the average of the ensemble. Is this a concern for the averaging? 

3. From algorithm 1, in the inner loop where generator is being updated, the global model's parameter theta_S is fixed. This would affect the quality of loss terms in equation (5) because the global model is still bad. Could the author comment on this issue?

4. In global model training (distillation), the author uses KL divergence as objective (equation (6)). Have the authors tried other losses such as cross entropy? 


limitations: The author has described limitations of current work and suggest for future investigation. They have not discussed negative societal impact, which I believe is OK.
review_title: null

Meta-review: metareview: This work proposes a new one-shot FL algorithm. It consists of two steps on the server: a data generation step that trains a GAN to synthesize data utilizing the local models and a distillation step that distills the ensembles local models using the generated data. The method has several advantages in comparison with other one-shot FL algorithms. The performance is verified by experiments. 

One major concern in the reviews was regarding novelty. This has been addressed by the author. 

Please clarify the following in the final version
1. Teacher's model: What is the quality of the ensemble model (teacher) in the experiment? Does the distilled model improve over the teacher (similar to self-distillation)? Showing the distillation gap is important to understand how the method works.
2. Contribution of GAN in quality: From pure quality point of view, what if the original data is used to train the ensemble and distilled models?

Please also consider adding privacy-utility trade offs in the future work. It is true that one-shot FL is in general more secure than multi-round methods, and some DP work can be applied here directly. But showing on-par or better privacy-utility trade off is an important justification on why it should be adopted.
confidence: Less certain
Decision: Accept