FedL2G: Learning to Guide Local Training in Heterogeneous Federated Learning

Authors: Jianqing Zhang, Yang Liu, Yang Hua, Jian Cao, Qiang Yang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Heterogeneous Federated Learning, Model Heterogeneity

Abstract:
Data and model heterogeneity are two core issues in Heterogeneous Federated Learning (HtFL). In scenarios with heterogeneous model architectures, aggregating model parameters becomes infeasible, leading to the use of prototypes (i.e., class representative feature vectors) for aggregation and guidance. However, they still experience a mismatch between the extra guiding objective and the client's original local objective when aligned with global prototypes. Thus, we propose a Federated Learning-to-Guide (FedL2G) method that adaptively learns to guide local training in a federated manner and ensures the extra guidance is beneficial to clients’ original tasks. With theoretical guarantees, FedL2G efficiently implements the learning-to-guide process using only first-order derivatives w.r.t. model parameters and achieves a non-convex convergence rate of $\mathcal{O}(1/T)$. We conduct extensive experiments on two data heterogeneity and six model heterogeneity settings using 14 heterogeneous model architectures (e.g., CNNs and ViTs) to demonstrate FedL2G’s superior performance compared to six counterparts.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729426007951

thread_type:
official_review

<thread_object>
creator: reviewer_uwko

<content>
summary:
This paper focuses on two classic challenges in Federated Learning (FL): data heterogeneity and model heterogeneity, both of which are crucial for deploying FL in real-world scenarios. In model-heterogeneous settings, aggregating model parameters is not feasible, making the aggregation of class prototypes a common approach. However, the authors observe that the simple aggregation of locally uploaded class prototypes into a global prototype fails to effectively guide the local training process. Thus, the authors propose a Federated Learning-to-Guide (FedL2G) method, which adaptively learns to guide local training in a federated manner and ensures that the additional guidance is beneficial to clients’ original tasks.

strengths:
1) The paper tackles two critical challenges for deploying Federated Learning: data heterogeneity and model heterogeneity, which are essential for real-world FL deployment.
2) The authors identify that simple aggregation of class prototypes can even harm local updates, and therefore propose a learnable approach to generate global prototypes that effectively guide local training processes.
3) The paper provides a convergence proof, validating the theoretical feasibility of the proposed method.
4) Extensive experiments are conducted to demonstrate the effectiveness of FedL2G.

weaknesses:
1) The meaning conveyed by Figure 1 is unclear. It lacks sufficient explanation, making it difficult to understand the challenges the paper aims to address based on the illustration.
2) Figure 2 has poor readability, and the lines are overly complicated and cluttered, making it hard to capture the key points. I recommend adding some descriptions to aid in understanding.
3) Although Figure 1 somewhat points out the challenges to be solved, the paper lacks sufficient motivation to validate this phenomenon.
4) I am concerned about whether uploading local class prototypes or features of local data samples may leak local data privacy since these data represent local statistical information.
5) There is a misuse of symbols. In Equation 1, the denominator uses "n" to represent the overall data, which may be misunderstood as "n" representing a specific device, especially since there are a total of N clients.
6) There is a lack of qualitative or quantitative analysis to prove that the proposed method effectively reduces the cross-entropy loss on users' local data.
7) Equation 6 aggregates gradients uploaded by all clients in a simple manner, which I believe may further introduce bias. This is because the information uploaded by strong clients (with stronger feature extraction capabilities and more data) should be more valuable than that from weaker clients.

questions:
1) For the prototypes uploaded by different clients, can dynamic aggregation be performed based on the performance of each client? Strong clients often have richer information, so the initial global prototype could be aggregated based on their information to guide the training of weaker clients. As weaker clients produce more robust prototypes, they can also upload their local class prototypes to aggregate a new global prototype.
2) For other issues, see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729557503842

thread_type:
official_review

<thread_object>
creator: reviewer_dhdr

<content>
summary:
To address the objective mismatch in Heterogeneous Federated Learning, this paper proposes a Federated Learning-to-Guide method that adaptively learns to guide local training in a federated manner and ensures the extra guidance is beneficial to clients’ original tasks. The technique efficiently implements the learning-to-guide process using only first-order derivatives and achieves a non-convex convergence rate of $\mathcal{O}(1/T)$. The authors provide empirical validations on the theoretical results as well.

strengths:
The proposed method is well-motivated, the paper shows that existing methods suffer from the objective mismatch issue, and show how to fix it.

The empirical results show that Federated Learning-to-Guide method is better than other lightweight HtFL methods, as expected, and the authors conduct ablations that show the influence of the server learning rate and visualizations of data distributions.

weaknesses:
The analysis only covers FedOpt with SGD as the optimizer. Still, recent work [1] shows a different combination (Nesterov-accelerated SGD as the outer optimizer and AdamW as the inner optimizer) is much better for practical performance.

In the theorems that are presented, summarizing the main insights of these theorems may be needed since currently they are just written as long paragraphs.

In experiments, the least partial client participation ratio is set as 0.5. In more realistic settings, the participation ratio is lower with more clients.

[1].	Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam, Jiajun Shen. DiLoCo: Distributed Low-Communication Training of Language Models. arXiv:2311.08105.

questions:
See in weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652016736

thread_type:
official_review

<thread_object>
creator: reviewer_xlkd

<content>
summary:
The paper under review studies heterogeneous federated learning (HtFL) and aims to address the mismatch between local and global learning objectives. By learning a set of local guiding vectors during model training, the local loss is prioritized when minimizing the global loss. The guiding vectors are compact in size and are updated based on feedback from clients’ local quiz sets. Therefore, the proposed FedL2G method is claimed to be lightweight, efficient, and adaptable.

strengths:
1. Sections 1 and 2 clearly introduce the research problem, relevant literature, and contributions, making the paper easy to follow.

2. The mismatch between model personalization and generalization is identified as a primary research problem in the field of HtFL. 

3. Guiding vectors, which are compact in size, are communicated between clients, thereby reducing communication overhead compared to direct model sharing.

4. Relevant benchmarks and theoretical analysis are included to support the paper’s contributions.

weaknesses:
1. Feature Extraction Consistency: The consistency of feature extraction is not discussed in detail. With heterogeneous models and quiz sets across different clients, distributed feature extraction may produce varying representations (i.e., guiding vectors) for the same class. How can the averaging of vectors in line 13 of Algorithm 1 be effective in this context? Are there any explanations of feature extraction consistency when local models and private datasets are heterogeneous? 

2. Warm-Up Period in Large-Scale Distributed Learning: In practical settings, a warm-up period may not be feasible in large-scale distributed learning scenarios. Does the warm-up phase require all clients to join the system for at least 50 rounds, as indicated on line 236? An ablation experiment should be added to help readers understand why the warm-up is necessary or to quantify the accuracy loss if it is omitted. 

3. Data Heterogeneity Settings: The data heterogeneity settings may be overly strict. Using a Dirichlet distribution with control parameters like 0.1 and 0.01 makes the dataset highly skewed. The authors should explore a range of settings, from 0.01 to 1, to show how data heterogeneity impacts the performance of FedL2G. 

4. Quiz Set Requirement: The proposed approach assumes that each client can maintain a quiz set, which implies that clients have sufficient training data. The authors should clarify a practical scenario to support this assumption and ensure that the settings are realistic.

questions:
1. How can the averaging of vectors in line 13 of Algorithm 1 be ensured to be effective? Are there any explanations regarding feature extraction consistency when local models and private datasets are heterogeneous? The authors should provide quantitative analysis to assess how the consistency of feature extraction impacts the performance of the guiding vectors, for example, measuring cosine similarity between guiding vectors from different clients for the same classes, or examining how variance in feature representations correlates with model performance.

2. Does the warm-up phase require all clients to join the system for at least 50 rounds, as indicated on line 236? An ablation experiment should be included to help readers understand the necessity of the warm-up phase or to quantify the accuracy loss if it is omitted. The review suggests the authors compare performance with different warm-up durations (e.g., 0, 25, 50, 100 rounds) and analyze how it affects convergence speed and final accuracy across various client participation scenarios.

3. Will the proposed method outperform benchmarks in both homogeneous and heterogeneous data settings? The authors should explore a range of heterogeneity settings, with a verity of Dirichlet parameters, to demonstrate how data heterogeneity impacts the performance of FedL2G. The reviewer suggests adopting specific Dirichlet parameter values to test (e.g., α = 0.01, 0.1, 0.5, 1, 10) and suggests key metrics to report for each setting, such as accuracy, convergence speed, and communication efficiency.

4. What real-world scenario allows for sufficient quiz sets and warm-up phases? The authors should clearly state a practical scenario to justify the feasibility of these settings. The reviewer suggests that the authors may provide specific examples of real-world applications where these requirements could be met, or discuss potential modifications to the method for scenarios with limited data or stricter time constraints.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688723594

thread_type:
official_review

<thread_object>
creator: reviewer_utfs

<content>
summary:
The paper introduces the FedL2G method, designed to address challenges in Heterogeneous Federated Learning that arise from data and model heterogeneity. The proposed method focuses on optimizing the training process by ensuring that additional guiding objectives introduced during the federated learning process are beneficial and align with clients' original local objectives.

strengths:
1. The paper is well-written and easy to follow.
2. The paper is well-motivated that Heterogeneous Federated Learning is quite realistic.
3. The authors provide rigorous convergence analysis.

weaknesses:
1. The paper may overlook some important related methodologies, such as FedPCL[1] and FPL[2], which also address the central issue highlighted by the authors: the deviation of aggregated global prototypes from client-specific feature vectors due to data heterogeneity. Specifically, FPL tackles similar challenges, making its comparison with FedL2G pertinent for a comprehensive evaluation.
2. The proposed method introduces complexity by requiring an additional small quiz set and a warm-up period, which could complicate implementation. The quiz set, while beneficial for validating model updates, may also introduce an unfair advantage over other baselines that do not use this approach.
3. It would be beneficial for the authors to include more recent federated learning methodologies like FedPCL[1] and FPL[2] in the experimental comparisons. The current baseline methods, while foundational, may not represent the state-of-the-art, limiting the robustness of the comparative analysis presented.
4. Including experiments on the DominNet dataset could enhance the applicability and robustness of the FedL2G method across diverse scenarios. 

[1] Tan, Y., Long, G., Ma, J., Liu, L., Zhou, T., Jiang, J.: Federated learning from pre-trained models: A contrastive learning approach. Advances in Neural Information Processing Systems 35, 19332–19344 (2022)

[2] Huang, W., Ye, M., Shi, Z., Li, H., Du, B.: Rethinking federated learning with domain shift: A prototype view. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16312–16322. IEEE (2023)

questions:
Please see the weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732601004096

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you for your valuable feedback and insights

comment:
Dear Reviewers,  

Thank you for your dedicated time and effort in reviewing our work. As the discussion phase nears its end, we kindly invite your feedback on our responses to ensure we’ve thoroughly addressed your concerns.  

If you have any additional questions or points for discussion, we’d be happy to engage further. Your input is invaluable, and we deeply appreciate your consideration.  

Best regards,  
The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732945614001

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Sincere Thanks for Reviewers' Constructive Feedback

comment:
Dear Reviewers,

We would like to express our heartfelt gratitude for the time and effort you have dedicated to reviewing our paper. Your thoughtful comments (**mainly focused on adding auxiliary experiments for further improvement**) have been invaluable in improving both the clarity and quality of our work. For your convenience, we’ve summarized our rebuttal below.

### **Contributions of our Paper**

1. We observed a negative effect—an increase in the **original local loss**—resulting from the use of an additional **guiding objective** during each client’s local training in representative prototype-based methods in HtFL, particularly under data and model heterogeneity. We analyzed this phenomenon and attributed it to a **mismatch** between the prototype-guiding objective and the client's original local objective.
2. To address the objective mismatch issue, we propose a novel HtFL method, FedL2G, which **prioritizes the original local objective while incorporating the guiding objective**. Instead of using the extracted and aggregated feature vectors (i.e., prototypes) in the guiding objective, we learn the guiding vectors from scratch using a meta-learning approach. In other words, **the semantics of prototypes and guiding vectors differ**. While prototypes are feature vectors, **guiding vectors are not**—they may lie outside the feature space of the client models. **The semantics of guiding vectors can be understood as "learned vectors designed to further reduce the original local loss."** This is **a common expectation among all clients, making it worthwhile to share.**
3. We provide **theoretical guarantees** and efficiently implement the learning-to-guide process using only first-order derivatives with respect to model parameters. This approach achieves a **non-convex convergence rate of O(1/T)**.
4. We conduct experiments across **two data heterogeneity** and **six model heterogeneity** settings, enabling collaborative learning across **14 heterogeneous model architectures (e.g., ResNets and ViTs)** within a single FL system. Our method demonstrates superior performance compared to six heterogeneous-model-applicable state-of-the-art approaches.


### **Summary of our rebuttal**

1. **Experiments with related approaches.** We included a comprehensive set of widely used heterogeneous-model-applicable state-of-the-art baselines in our experiments at the time of submission. During the rebuttal, we also compared our method with the suggested homogeneous-model approaches, FedPCL and FPL, as well as a newly accepted method, FedMRL.
2. **Experiments on more datasets and settings.** During the rebuttal, we included additional experiments on DomainNet, exploring different settings for the warm-up round (T'), varying degrees of data heterogeneity (β), different quiz set sizes (qss), and varying client participation ratios (ρ) across 100 clients. We also introduced noise to further enhance privacy, with negligible impact on performance.
3. **Further clarification on the quiz set and warm-up period.** The **quiz set is not an additional dataset** but a small portion held out from the original training data (see Lines 65-67), ensuring fairness without providing any extra data advantage over other baselines. **Only 2 to 5 samples are sufficient** for our FedL2G to achieve strong performance. The warm-up phase (steps 1, 3, 4, 5, 6, 7) is computationally lightweight and mirrors the main FL process, except for skipping step 2 (local model updates) (see Lines 233-240), requiring minimal additional effort. Furthermore, our FedL2G-l achieves an accuracy of **41.7 with (T' = 0) (i.e., no warm-up)**, while FedL2G-f achieves **41.6 with (T' = 1)** on Cifar100 in the default Dirichlet setting using HtFE$_8$.
4. **Improve the readability of Figures 1 and 2, as well as the notations.** We thank the reviewers for their constructive feedback on readability, and we have revised the paper accordingly in the newly uploaded version.

If you have any further concerns, we would be more than happy to address them :-).

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces the FedL2G method, designed to address challenges in Heterogeneous Federated Learning that arise from data and model heterogeneity. The proposed method has theoretical guarantees, and have been shown to outperform prior works. 

While the approach is supported by theoretical guarantees, the strength of these results is limited. This is because a non-collaborative setting could achieve similar guarantees. It is recommended to include such a baseline and provide a comparative discussion in the revision. A promising direction for future work would be to demonstrate convergence speedup as the number of agents increases, akin to the linear speedup observed with homogeneous agents. Additionally, it may be valuable to introduce a parameter to quantify agent heterogeneity, showing that as agents become more homogeneous, the convergence rate improves.