Title: FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Data Heterogeneity

Statistical Rectification
Abstract: Federated Learning (FL) confronts a significant challenge known as data heterogeneity, which impairs model performance and convergence. Existing methods have made notable progress in addressing this issue. However, improving performance in certain heterogeneity scenarios remains an overlooked question: _How robust are these methods to deploy under diverse heterogeneity scenarios?_ To answer this, we conduct comprehensive evaluations across varied heterogeneity scenarios, showing that most existing methods exhibit limited robustness. Meanwhile, insights from these experiments highlight that sharing statistical information can mitigate heterogeneity by enabling clients to update with a global perspective. Motivated by this, we propose **FedGPS** (**Fed**erated **G**oal-**P**ath **S**ynergy), a novel framework that seamlessly integrates statistical distribution and gradient information from others. Specifically, FedGPS statically modifies each client’s learning objective to implicitly model the global data distribution using surrogate information, while dynamically adjusting local update directions with gradient information from other clients at each round. Extensive experiments show that FedGPS outperforms state-of-the-art methods across diverse heterogeneity scenarios, validating its effectiveness and robustness. The code is available at: <https://github.com/CUHK-AIM-Group/FedGPS>.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a federated learning framework, named FedGPS, to address the data heterogeneous problems in FL, which 1) uses surrogate information to guide the local models to learn data distribution aligned with the global distribution and 2) refines the update direction at each step based on other client information at the gradient view.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n* S1: The problem solved in this paper, data heterogeneity in federated learning, is an important and widely concerned issue.\n\n* S2: The paper designs the method from the perspective of loss-landscape, which has good interpretability.\n\n### Weakness\n\n* W1. The paper lacks novelty and is incremental. The method proposed in the paper seems to be a combination of the method proposed in [14,23] and the method proposed in [24].\n\n* W2. Related work is incomplete and lacks some key SOTA works. For example. lack of comparison with methods based on heuristic search [R1][R2]. The motivation of these methods is similar to SAM, aiming to improve the generalization of FL by guiding training towards flat areas. Since these methods use completely different technical routes from SAM, it is incomplete for the author to only mention SAM in related work, and these methods need to be discussed and compared. In addition, discussing more the SOTA SAM-based works [R3][R4] can improve the quality of the paper.\n\n* W3. From the training curves (Figures 4, 7, 10, and 11) provided by the author, the performance improvement seems limited. In addition, the author uses the highest accuracy as the evaluation standard of accuracy in the table, but it is difficult to accurately stop the training at the round with the highest performance in the real scene. I suggest that the author use the mean and standard deviation to show the experimental results.\n\n* W4. The authors seem to have only evaluated extreme heterogeneous scenarios, not normal heterogeneous scenarios, such as \\alpha=0.3, 0.6, and IID, which are commonly used settings.\n\n* W5. The author's experiments on cross-device scenarios are insufficient, lacking experiments on more clients (e.g., K=500, 1000) and different participation ratios (the author only evaluates 10% of devices participating in local training).\n\n* W6. It is unreasonable for the author to set E=1 for the main experiments, which will cause the FL communication overhead to increase sharply. In addition, according to the experimental results in the appendix, when E=5, the advantage of the method decreases significantly, and the author needs to give an explanation.\n\n[R1] Is aggregation the only choice? Federated learning via layer-wise model recombination. KDD 2024\n\n[R2] FedCross: Towards accurate federated learning via multi-model cross-aggregation. ICDE 2024\n\n[R3] Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning. CVPR 2025\n\n[R4] Flexible Sharpness-Aware Personalized Federated Learning. AAAI 2025\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first identified a key issue that many existing FL algorithms lack robustness across diverse data heterogeneity scenarios. To address this problem, Federated Goal-Path Synergy (FedGPS) is proposed, aiming to enhance the robustness by leveraging both statistical and gradient information from other clients. Specifically, the author proposed a static goal-oriented objective function, including two stages: (1) aligning local distributions with a local surrogate distribution, and (2) aligning the local surrogate distribution with the global surrogate distribution. Another technique, dynamic path-oriented gradient rectification, is proposed to integrate non-self gradient information into local updates to correct the local optimization directions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper is well-motivated and clearly structured. \n- The idea of modeling the global data distribution using a two-stage surrogate distribution alignment is novel and intuitive.\n- The authors conducted extensive and thorough experiments under a wide range of non-IID conditions to demonstrate the robustness of their proposed FedGPS compared to comprehensive baseline approaches.\n\nWeaknesses:\n- I am missing some details on how exactly the surrogate distribution is generated. The paper only provides a high-level description: “sampled from distinct Gaussian distributions”.\n- The paper lacks sufficient discussion on any potential privacy risks from their proposed two techniques. The authors claim that the surrogate distribution is a privacy-preserving approach since no raw statistical information is shared. However, the local surrogate distributions might still be vulnerable to reconstruction or membership inference attacks.  Additionally, the dynamic path-oriented gradient rectification explicitly utilizes gradient information derived from other clients. More comprehensive analysis and clarification regarding the potential privacy risk associated with this gradient-sharing technique are needed.\n- The motivation provided in Section 4.3, specifically the statement, \"This insight builds on the ingenious concept that a malicious client in FL …, offering extra advantages,\" seems problematic and may require further clarification.\n\nQUESTIONS:\n1. Could the authors provide additional technical details regarding the generation process of surrogate distributions? Moreover, it would be beneficial to provide some justifications for why Gaussian distribution was chosen.\n2. How do the authors plan to address potential privacy concerns associated with the proposed two techniques (i.e., surrogate distributions and share of non-self gradient information)?\n3. Have the authors considered evaluating other non-iid scenarios beyond the Dirichlet partitioning method, such as assigning each client data from exactly C distinct classes (C=2 or C=3), ensuring that each client only sees a specific subset of all labels?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed FedGPS, a framework that integrates statistical distribution and gradient information from other clients in federated learning, aiming to mitigate data heterogeneity issues and improve the performance of trained models. The framework consists of a static goal-oriented objective that incorporates feature alignment between local real and surrogate distributions, as well as alignment between local and global surrogate distributions, and dynamic path-oriented rectification that rectifies local update direction using gradients from other clients. Experiments showed that the proposed method achieved improved accuracies and convergence speed compared to previous work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper proposed an effective method to tackle data heterogeneity issues. Statistical information when properly used can help alleviate the divergence between local and global models while preserving user privacy. The proposed method is intuitive and could be applied in a wide range of cases.\n\n2. The paper conducted extensive experiments to show the performance of the proposed method, which improves over previous work in most cases.\n\nWeaknesses:\n\n1. The paper did not formally define the local surrogate distribution and global surrogate distribution, which are crucial components of the proposed method. In addition, more information needs to be transmitted between the server and clients such as the global features and gradients of other clients. It should be made clear what information is transmitted in each round of communication.\n\n2. The notations are inconsistent in Sections 3 and 4. Below are some examples:\n\n- In line 111 the set of clients is K but in line 112 it is S.\n- In line 163, sum of distributions.\n- In lines 175-176, Stage 1 and Stage 2 are not further defined or referenced.\n- In line 180, \\delta in bound.\n\nQUESTIONS:\n1. Why is it necessary to introduce the surrogate distributions? Based on the assumption that the global surrogate distribution is computed from the global real distribution (which is not defined in the paper), if equation (6) is the objective function for local update, then why not simply use the real global and local statistics, i.e., use the distance between the local feature and global feature (sent from the server) in the loss function? In addition, what is purpose of aligning the local real distribution and the local surrogate distribution if the latter stems from the former one?\n\n2. What is the communication overhead introduced by the extra statistical and gradient information transmission?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
Federated Learning (FL) struggles with data heterogeneity, limiting model performance. Existing methods lack robustness across diverse scenarios. This work shows that sharing statistical information helps mitigate heterogeneity by providing a global perspective. Inspired by this, the authors propose ​​FedGPS (Federated Goal-Path Synergy)​​, a novel framework that combines statistical distribution and gradient information from other clients. FedGPS statically adjusts learning objectives to model global data distribution and dynamically refines local updates using external gradient information. Experiments demonstrate that FedGPS outperforms existing methods in various heterogeneity scenarios, proving its effectiveness and robustness across datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper proposes a robust approach (FedGPS) that effectively integrates statistical and gradient information to address data heterogeneity.
2. This paper conducts experiments across diverse datasets and heterogeneity scenarios.
3. This paper is easy to read and follow.


Weaknesses:
1. For Figure 1, could the authors provide more explanation about the Nemenyi post-hoc test? This method appears less commonly used in evaluations.
2. In line 79, the authors describe refining the update direction using other clients' information. Would it be possible to support this with theoretical justification?
3. For Equation 7, could the authors clarify:
- How is λ determined?
- Why does incorporating non-self gradients improve the update direction?
4. Have the authors considered evaluating their method on Vision Transformers (ViTs) or NLP tasks?

QUESTIONS:
Please refer to the weakness part.

LIMITATIONS:
Please refer to the weakness part.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work conducts comprehensive evaluations across different heterogeneous scenarios, and asserts most existing methods exhibit limited robustness. Motivated by the observation, this work proposes FedGPS, which statically modifies each client's learning objective to resemble the global data distribution, while adjusting local update directions with additional gradient information.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength:\n- The paper is well-written and easy to follow. \n- Reported empirical performance is strong, although the reported metrics seem a bit unfair, if the communication cost of non-self gradient computation is not taken into account.\n\n### Weakness:\n- It seems that the proposed method requires many more communications than existing methods. In particular, I'm confused about the non-self gradient for each client. It seems that each client has to communicate high-dimensional vector to every other client. This is even more expensive than one \"communication round\" of federated learning in the traditional sense (every participating client periodically communicates with the server). Is this true?\n\nQUESTIONS:\n- Doesn’t incorporating gradient information from other clients incur additional computational cost? In Def 4.2, it seems that to compute $\\delta_{\\theta_i}^t$, each participating client has to communicate $\\Delta_{\\theta_k}^{t-1}$. Is this incorporated as \"communication round\" in the experiments? \n\n- By communicating local gradient information with other clients, can it hurt the improved privacy aspect of federated learning? \n\n- How come comparison to adaptive methods [1, 2, 3] are not included? These also try to address heterogeneity issue in federated learning.\n\n[1] S. Reddi et al., Adaptive Federated Optimization\n\n[2] J.L. Kim et al., Adaptive Federated Learning with Auto-Tuned Clients\n\n[3] S. Mukherjee et al., Locally Adaptive Federated Learning\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}