FedGO : Federated Ensemble Distillation with GAN-based Optimality

Authors: Won-Jun Jang, Hyeon-Seo Park, Si-Hyeon Lee
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Federated learning, ensemble distillation, data heterogeneity, generative adversarial network

Abstract:
For federated learning in practical settings, a significant challenge is the considerable diversity of data across clients. To tackle this data heterogeneity issue, it has been recognized that federated ensemble distillation is effective. Federated ensemble distillation requires an unlabeled dataset on the server, which could either be an extra dataset the server already possesses or a dataset generated by training a generator through a data-free approach. Then, it proceeds by generating pseudo-labels for the unlabeled data based on the predictions of client models and training the server model using this pseudo-labeled dataset. Consequently, the efficacy of ensemble distillation hinges on the quality  of these pseudo-labels, which, in turn, poses a challenge of appropriately assigning weights to client predictions for each data point, particularly in scenarios with data heterogeneity. In this work, we suggest a provably near-optimal weighting method for federated ensemble distillation, inspired by theoretical results in generative adversarial networks (GANs). Our weighting method utilizes client discriminators, trained at the clients based on a generator distributed from the server and their own datasets. 
Our comprehensive experiments on various image classification tasks illustrate that our method significantly improves the performance over baselines, under various scenarios with and without extra server dataset. Furthermore, we provide an extensive analysis of additional communication cost, privacy leakage, and computational burden caused by our weighting method.

Decision: reject
Reviewer scores: 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730646675419

thread_type:
official_review

<thread_object>
creator: reviewer_kvtq

<content>
summary:
This paper proposes FedGO: Ferated Ensemble Distillation with GAN-based Optimality, for federated ensemble distillation. This algorithm incorporates a novel weighting method using the client discriminators that are trained at the clients based on the generator distributed from the server and their own datasets. The generator distributed from the server can be either off-the-shelf or trained with the unlabeled dataset on the server. The exchange of the generator and the client discriminators between the server and the clients occurs only once before the main FL algorithm starts, resulting in minimal additional overhead.
Extensive experiments demonstrate significant improvements of FedEDG over existing research both in final performance and convergence speed on multiple image datasets.

strengths:
1. The paper is well-written and easy to follow.
2. The authors conducted extensive experiments to verify the effectiveness of the proposed method.

weaknesses:
As far as I am concerned, distillation-based FL is data dependent and requires access to an auxiliary dataset derived from publicly available proxy data sources for knowledge transfer, whereas a desirable auxiliary dataset is not always available since its construction requires careful deliberation and even prior knowledge about clients’ private data to achieve satisfactory performance, which is inconsistent with the privacy-preserving nature of FL. In addition, I argue that FedEDG with Pretrained Generator proposed in this paper also has the above-mentioned issues. This is because pre-trained generator needs to be trained on public datasets. Therefore, I remain skeptical of this research direction, even if the paper contains theoretical evidence. Furthermore, if the author wants to convince me, please provide some feasible solutions to address the aforementioned issues.
I'll raise my score if author can address the above problems.

questions:
Please see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662552486

thread_type:
official_review

<thread_object>
creator: reviewer_5yyt

<content>
summary:
The paper introduces a new approach to address the issue of data heterogeneity in federated learning. By applying Generative Adversarial Network (GAN) techniques to federated ensemble distillation, the paper proposes a near-optimal weighting method that enhances the training process of the server model. Extensive experimental validation demonstrates significant improvements in model performance and convergence speed across various image classification tasks. Moreover, the study provides an in-depth analysis of the potential additional communication costs, privacy leaks, and computational burdens introduced by this method, showcasing its practicality and flexibility in protecting data privacy and enhancing system efficiency.

strengths:
This paper demonstrates originality through its innovative integration of GAN-based techniques with federated ensemble distillation. The use of discriminators trained at the client side to optimize the weighting of client contributions during the distillation process is a novel approach that has not been extensively explored in previous federated learning research.

The method's originality is further enhanced by its theoretical grounding, which employs results from GAN literature to develop a provably near-optimal weighting method. 

The experimental setup is well thought out

weaknesses:
The paper claims near-optimal performance based on theoretical justifications rooted in GAN literature. However, these claims might depend heavily on certain idealized assumptions about data distributions and discriminator performance. Real-world deviations from these assumptions could lead to suboptimal performance. The paper does not explain how to select discriminator architectures.

questions:
1. In introduction on page 2, Our main contributions are summarized in the following: "Federated Ensemble Distillation" instead of "Ferated Ensemble Distillation".

2. In theoretical analysis, near-optimal performance is heavily affected on discriminator performance. I do not understand how to select the discriminator architectures? Can you give me some detailed description?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691065975

thread_type:
official_review

<thread_object>
creator: reviewer_gtch

<content>
summary:
This paper proposed a novel federated ensemble distillation approach that utilizes generative adversarial networks (GANs) to address the challenges posed by data diversity across clients. Specifically, the proposed approach employs GANs to optimize the weighting of client predictions, thereby improving the quality of pseudo-labels generated during the ensemble distillation process. The paper provides theoretical insights that establish the effectiveness of the proposed method. Comprehensive experiments demonstrate that the proposed approach outperforms existing methods in robustness against data heterogeneity.

strengths:
1. The paper provides a theoretical foundation for the proposed approach, which validates its effectiveness and enhances its credibility.
2. The paper analyzes communication, privacy, and computational complexity within different scenarios, providing valuable insights for implementing the proposed approach.

weaknesses:
1. This paper needs to demonstrate the effectiveness of the proposed approach on different model structures, such as VGG and MobileNet.
2. The effectiveness of the proposed method relies on the quality of the discriminator and generator. The paper needs to conduct related ablation studies.
3. This paper should conduct ablation studies to analyze the impact of hyperparameters (e.g. $E_s$ and $E_d$) on the effectiveness of the approach.
4. The experimental settings of the baselines are not clearly stated, and it is important to clarify the fairness of the experimental comparison.
5. The additional computational and communication overhead introduced by the GAN-based approach may not be suitable for FL scenarios, particularly those with strict resource constraints.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732883246405

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We have uploaded a revised version of the paper, incorporating additional experiments and analyses conducted during the rebuttal period. All changes from the initially submitted paper are highlighted in blue. A summary of major changes is provided below:

1. **Additional Experiments with Alternative Architectures**
    
    **Classifier Architectures:** We conducted the main experiments using two different classifier architectures. The results demonstrate that FedGO consistently outperforms other baseline algorithms regardless of the model architecture. Details can be found in Appendix F.3.
    
    **Discriminator Architecture:** We assessed the impact of discriminator architectures on FedGO's performance. The results indicate that FedGO achieves similar final performance regardless of the discriminator architecture. These findings are summarized in Appendix F.6.
    
2. **Analysis of Impact of Hyperparameters**
    
    **Server Model Training Epochs:** We evaluated the performance of FedGO with varying numbers of training epochs for the server model. The experiments show that FedGO achieves higher performance than baseline algorithms even with fewer server model training epochs. This analysis is detailed in Appendix F.5.
    
    **Learning Rate Decay:** We investigated the effect of learning rate decay during server model training. The results reveal that FedGO performs better without learning rate decay. This is also discussed in Appendix F.5.
    
    **Generator Training Epochs:** We examined the effect of varying the training epochs of the generator on FedGO's performance. Interestingly, even when using an untrained, randomly initialized generator, FedGO outperforms or matches the performance of baseline algorithms. Further details are available in Appendix F.6.
    
3. **Additional Explanation on the Overhead of FedGO**
    
    We have provided a more detailed explanation emphasizing that the additional communication and computation overhead of our FedGO is minimal compared to previous federated ensemble distillation methods.
    

These updates aim to further clarify and strengthen the findings of our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310710276

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you for acknowledging that our weighting method for federated ensemble distillation is a novel approach with theoretically guaranteed optimality. However, concerns regarding additional costs due to the use of GAN have been raised. As we have explained in detail to the first and third reviewers, we would like to reaffirm that these additional costs are not a significant concern.

As detailed in Appendix G and our most recent response to the first reviewer (**Gtch)**, **FedGO incurs only negligible additional computational and communication costs on the client side, even in a fully data-free setup**. Despite this minor overhead, FedGO leverages a theoretically guaranteed, provably near-optimal weighting approach, enabling it to achieve state-of-the-art performance in federated ensemble distillation. Furthermore, it achieves comparable performance with fewer communication rounds, reducing client-side computation, communication, and privacy overheads through faster convergence. 

We respectfully assert that concerns about additional costs should not detract from the substantial contributions our work makes to the field of federated ensemble distillation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: The paper introduces FedGO, a federated learning approach that uses Generative Adversarial Networks (GANs) to optimize client prediction weighting in the ensemble distillation process, aiming to improve robustness against data heterogeneity. The approach provides theoretical insights and shows improved performance and convergence speed in image classification tasks. However, concerns remain about the additional computational overhead and privacy implications of FedGO.

Strengths:

FedGO offers a novel integration of GAN techniques with federated ensemble distillation, potentially enhancing the training process of server models.

The paper provides a theoretical foundation for the method and demonstrates its effectiveness through extensive experiments.
Drawbacks:

The approach may introduce significant additional computational and communication overhead due to the need for discriminator training and uploading, which could be prohibitive in resource-constrained federated learning scenarios.

There are concerns about the privacy implications of FedGO, as it requires the upload of locally trained discriminators, increasing the risk of privacy leakage.

The effectiveness of FedGO relies heavily on the quality of the discriminator and generator, and the paper lacks ablation studies to analyze the impact of hyperparameters and the robustness of the method under different conditions.

Given the above points, I must reject this work due to concerns about its practical applicability, the potential increase in privacy risks, and the need for more comprehensive analysis to address the drawbacks identified.