FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning
Authors: Zhiqin Yang
Yonggang Zhang
Chenxin Li
Yiu-ming Cheung
Bo Han
Yixuan Yuan
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Data Heterogeneity
Statistical Rectification
Primary area: deep_learning

Abstract: Federated Learning (FL) confronts a significant challenge known as data heterogeneity, which impairs model performance and convergence. Existing methods have made notable progress in addressing this issue. However, improving performance in certain heterogeneity scenarios remains an overlooked question: _How robust are these methods to deploy under diverse heterogeneity scenarios?_ To answer this, we conduct comprehensive evaluations across varied heterogeneity scenarios, showing that most existing methods exhibit limited robustness. Meanwhile, insights from these experiments highlight that sharing statistical information can mitigate heterogeneity by enabling clients to update with a global perspective. Motivated by this, we propose **FedGPS** (**Fed**erated **G**oal-**P**ath **S**ynergy), a novel framework that seamlessly integrates statistical distribution and gradient information from others. Specifically, FedGPS statically modifies each client’s learning objective to implicitly model the global data distribution using surrogate information, while dynamically adjusting local update directions with gradient information from other clients at each round. Extensive experiments show that FedGPS outperforms state-of-the-art methods across diverse heterogeneity scenarios, validating its effectiveness and robustness. The code is available at: <https://github.com/CUHK-AIM-Group/FedGPS>.

FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning
Zhiqin Yang' Yonggang Zhang
Chenxin Li
Yiu-ming Cheung Bo Han
Yixuan Yuan
The Chinese University of Hong Kong Hong Kong Baptist University The Hong Kong University of Science and Technology
Abstract
Federated Learning (FL) confronts a significant challenge known data hetero- geneity, which impairs model performance and convergence. Existing methods have made notable progress in addressing this issue. However, improving perfor- mance in certain heterogeneity scenarios remains overlooked question: How robust are these methods deploy under diverse heterogeneity scenarios? To answer this, we conduct comprehensive evaluations across varied heterogeneity scenarios, showing that most existing methods exhibit limited robustness. Mcan- while, insights from these experiments highlight that sharing statistical information can mitigate heterogeneity by enabling clients to update with a global perspec- tive. Motivated by this, propose FedGPS (Federated Goal-Path Synergy), novel framework that seamlessly integrates statistical distribution and gradient information from others. Specifically, FedGPS statically modifies each client's s learning objective to implicitly model the global data distribution using surro- gate information, while dynamically adjusting local update directions with gra- dient information from other clients at each round. Extensive experiments show that FedGPS outperforms state-of- of-the-art methods across diverse heterogeneity scenarios, validating its effectiveness and robustness. The code is available at: https://githut. com CUHK- AIM- Group/FedPS...
Introduction
Federated Learning (FL) facilitates collaborative model training across distributed data sources, garnering substantial interest recent years [1, 2, 3, 4]. Its primary goal is to keep data localized to protect sensitive information while harnessing contributions from other participants enhance individual models [5, 6] or construct a superior global model [7]. However, this decentralized paradigm encounters data heterogeneity [8, 9] (also known statistical heterogeneity), due the variations in client devices, geographic locations, and annotation processes [10, 11j. This departure from the assumption of independent and identically distributed (i.i.d.) data presents substantial challenge, complicating the training of distributed networks across diverse data distributions to achieve robust generalization on the overall data distribution [8, 12].
To enhance performance in FL, numerous studies have advanced efforts to mitigate the impact of data heterogeneity [13, 14, 15, 16]. FedAvg [7] introduces the paradigm of local training followed by aggregation. Moreover, several studies have refined the learning objective local training by incorporating constraints to mitigate client drift [11, 12, 17, 18]. Client sampling [19, 20] and global aggregation weight adjustments [21, 22] have also been tailored to adapt to heterogeneity scenarios. Additionally, information-sharing strategies [23, 14, 15] have emerged as promising approach
"Corresooning author: Yixuan Yuan (yxyuan www..cuk..c...ik.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

16
PoOLESAMS
RodDisco
Putttp
VHL Red
MUGFL
FOOLESONS
Rediap
instughh
Ted
Tecihse
SCAFOOLD
redirox
Fedbacar
TedRoggh
(a) SVHN
(b) CIFAR-10
Figure I: Nemenyi post-hoc test results on the performance under (a) SVHN and (b) CIFAR-10. Black horizontal lines indicate the critical distance (CD).
mitigate heterogeneity, though they often require increased communication or computational resources and demand careful privacy considerations to protect sensitive data.
Distributed environments inherently complicated, leading diverse scenarios involving different clients. As result, dataset heterogeneity varies across settings. To assess the robustness of algorithms under various data distribution scenarios, we raise previously underexplored question:
To what extent do existing methods maintain robustness across diverse scenarios, and by what mechanisms this robustness achieved?
The results presented in Fig. 1, Tabs. 1, 2, 3 and 4 show that most methods exhibit limited robustness, as indicated by the CD intervals that overlap between methods. This overlap highlights the challenges these methods face in adapting diverse data distributions. Nevertheless, the findings indicate that statistical information from other clients provides valuable insights for refining local updates. Moreover, sharing detailed information risks privacy leakage, which contravenes the core principles of FL. while coarse-gras saaaneg statistics, such as CCVR [13]. sharing the mean and covariance of logits, offer only marginal improvements adaptability across varied scenarios. Thus, determining which statistical information to use and how to leverage it effectively remains a significant challenge.
First, revisit the objective of FL, wherein each client trains model on its local data distribution, and these models are aggregated with the expectation of achieving robust generalization across the global data distribution. However, this process often introduces distribution gap due to data heterogeneity. (1) Distribution-Level: Inspired by [14, 24), we introduce static modification to the goal of local training, enabling implicit learning of the global data distribution through privacy- free surrogate distribution via a two-stage statistical information alignment process, as depicted in Fig. 3(a). Stage 1, the local data distribution is aligned with local surrogate distribution using the local model. Stage 2, the local surrogate distribution is aligned with a global surrogate distribution. Through these stages, the divergence between the local and global distributions effectively bounded, improving generalization while maintaining privacy.
Furthermore, achieving effective distribution alignment becomes difficult when the distribution shift substantial or the amount of data available per round is limited (e.g., low client sampling rate). (2) Gradient-Levet. Drawing inspiration from [25], propose incorporating gradient information from other clients prior to determining the local update direction. This strategy highlights the importance of utilizing insights from other clients' gradients to dynamically adjust the local optimization path at each step, ensuring more globally consistent update direction. Moreover, theoretical analysis reveals that careful parameter tuning of this gradient term can further rectify the update direction, resulting in a measurable reduction in the global model's loss function. Building on this two- level alignment strategy, introduce FedGPS, synergistic framework that integrates goal and path coordination, designed to ensure robustness in aubel-distribution-annooo experiments conducted on three benchmark datasets confirm the effectiveness of FedGPS, showcasing its superior performance across diverse scenarios.
scenarios. Extensive
Our contributions summarized follows:
We comprehensively evaluate existing federated learning methods designed to address heterogeneity, showing that most exhibit limited robustness across diverse distribution partitions. Our findings highlight the significant potential of leveraging statistical information from other clients to enhance performance.

---

Motivated by these insights, we attempt to propose a new framework to adapt to various heterogeneity scenarios called FedGPS, which incorporates statistical information from other clients from two perspectives. At the distribution level, we constrain local models to learn data distribution aligned with the global distribution using surrogate information. Concurrently, we refine the update direction at each step based on other client information at the gradient view, enabling a more holistic optimization process.
Extensive experiments with framework across diverse settings and benchmark datasets demonstrate the efficacy of FedGPS. Our results show that FedGPS surpasses existing methods, achieving robust and SOTA performance across various distribution splits.
2 Related Work
Federated Learning (FI.) enables localized data processing to preserve sensitive information, but this often results in data heterogeneity due to diverse data collection conditions. To mitigate this, several strategies have been developed to align local optimization with global objectives. For instance, FedProx [11] incorporates proximal term limit divergence between local and global parameters, ensuring more stable updates. Similarly, SCAFFOLD [12] introduces a control variate to correct local updates, while PAdaMFed [18] enhances convergence by integrating gradients and control terms from consecutive rounds to better estimate the global optimization direction. Another promising approach focuses on achieving a flatter loss landscape to enhance model robustness against heterogeneity. Techniques such as FedSAM [26]. MoFedSAM [27], FedGAMMA [28], and FedLESAN [29] perturb local parameters before updates, improving generalization and robustness, as supported by sharpness-aware minimization principles [30]. Sharing information among participants has garnered increasing attention. FedProto [31] shares class prototypes instead of model parameters, preserving privacy while inspiring subsequent approaches such as FedProK [32] and PILORA [33]. Additionally, generative models [34, 35] and local statistical methods [13] have been effectively employed to address heterogeneity challenges, enhancing model robustness across diverse data distributions. However, these methods may raise additional privacy concerns, prompting exploration of srivacy-preserving mechanisms [36].
On the server side, optimizing client selection strategies [37, 38, 39] is crucial for minimizing communication overhead by prioritizing clients most relevant to the global model, thereby improving efficiency. Advanced aggregation techniques further address heterogeneity. FedDisco [21] employs discrepancy-awart weights that consider factors beyond mere data size, while other works revisit aggregation protocols for improved performance [40]. Some methods design different aggregation methods on the server side, such as FedMR [41]. Server-side generative approaches, such as data- free knowledge distillation [42, 43], have also been explored mitigate heterogeneity, offering a complementary perspective to client-side innovations. Besides, FL has also received lot of attention in many areas, e.g. healthcare [44, 45] and transportation [46].
3 Preliminary
Federated Learning. In a typical federated learning setup [7, 47J, data samples are distributed across of K participating clients S [1, 2..., K}. without being centralized server. Each client E S maintains a local model parameterized by 0₂ and collaboratively contributes to training a global model parameterized by O. For each client k; the i-th data sample Éx. (X Mk..),, is drawn from its private local distribution D;. Then, the federated learning process can thus be formulated as the following optimization problem:
K
0"
= arg min F(0) = Σ (0x),
0
where Ph represents the weight of client k. This equation captures the goal of Fl., which seeks to get the optimal global model 0* that minimizes the global objective F(O) by optimizing local objectives Fi (0x) for each client, expressed as:
Fx (02) = Eç [e(0x; Ex)l
(2)
where ((.,)) is the loss function, e.g., cross-entropy supervised learning task. The local update at t-th round usually follows the conventional stochastic gradient descent (SGD) with yy denoting the
3

---

Data Distribution Across Clicats
Data Distribution Across Clicats
distrbbuttoo geverg (lecel mnd gldbwi)
4000
-000
Class Labi
2500
Class
2000
1900
1000
1000
-0
proses
(a) Heterogeneous distribution
(b) Heterogeneous distribution (c) Distribution divergence measure
Figure 2: (a) and (b) are examples of data distribution scenarios generated using the Dirichlet partition method under the CIFAR-10 dataset across 10 clients. All scenarios use the same heterogeneity control factor of a = 0.1, but vary the random seed to produce different heterogeneous distributions. (c) The divergence between local and global surrogate distributions is computed the FL training proceeds with different ratios of client sampling, also means the proportion of the data that participates in the global update at each federated training round. The divergence is computed every 5 rounds.
local step size follows:
0 =
V Ex).
(3)
The locally updated models are uploaded to the server, which derives new global model through aggregation mechanism AGG(+) based the t-th round collected local data (c.g.. Eq(I)), global model 0¹, and the global step size 77g-
θ AGG(1g: 0"; (0)+ Tkes.),
(4)
where St denotes the set of clients participating the 1-th training round.
Definition 3.1 (Wasserstein Distance). Consider two probability distributions and the data space X x V, where X C. Rd is the feature space and y is the label space. Given a distance metric X V, the p- Wasserstein distance between mu and mu, for any p Z is defined as:
WP(nuv = (5 inf
1/p
I (ET(pp)) (z.g))pp.("" w'w
M(,, (dd w/) dx, (1,,
(x',
where TJ,, is the set of all joint distributions with marginals and v, respectively.
4 Methodology
This section details our proposed "FedGPS" framework. begin by outlining the motivation behind FedGPS(Sec. 4.1). Subsequently, we describe how statistical information is leveraged from two perspectives: the distribution view (Sec. 4.2) and the gradient perspective (See. 4.3).
4.1 Motivation
Performance degradation in FL stems from the divergence between local and global data distribu- tions. Training on shifted local distributions Dj while expecting generalization on the global i.i.d. distribution D, naturally creates distribution gap. This can be expressed as:
0" = arg min Ep, [F(AGG(O.)) where 0 = arg min Ep, [Fk, Ex)
(5)
Consequently, this divergence results in a performance gap with respect to the global distribution. The distribution shift can be quantified using the p- Wasserstein distance based Definition 3.1.
To address this gap, existing methods often share distribution
information. For example, FedProto [31] shares class-specific average embeddings prototypes, while FLGAN [48] uses synthetic data from Conditional GAN (CGAN) [49]. However, these approaches, which involve sharing data-derived information, introduce privacy risks. Additionally, VHL [14] employs

---

Caar
A
Clink
Chent
This direction better
Algrement glinal
Naargated global paoggtc Ritiitaiion
detritatio cuch
Cantuinci aliganorr
Aliganenn
direction
Orispiaal
path
Other elients
Clire? Cioii
I Leca min imaa
Privary
gradical
Global minima
Global deniiiiiiiiiio
Local dertbaatoon
(a) Static goal-oriented learning objective
Dynumic path-oriented gradient rectification
Figure 3: (a) Static goal-oriented objective. This objective is composed of two stages: local distri- bution aligns with local surrogate distribution (Alignment at local), and local surrogate distribution aligns with global surrogate distribution (Alignment with global). (b) Dynamic path-oriented rectifi- cation corrects the original update direction by the gradient of other clients for a new update path.
domain adaptation [50] with virtual homogeneity data, yet this data still exhibits distribution shifts, virtual data is trained shifted models at each client.
Drawing on our evaluations and prior work [14, 24], we introduce a sevaccypreserving surrogate distribution (e.g., sampled from distinct Gaussian distributions) to tackle the distribution gap in federated learning (FL). This surrogate improves FL performance by minimizing the upper bound of the global model's generalization error through two-stage alignment process. However, alignment quality varies with the volume of training data per round. For instance, in low client participation scenarios involving approximately 10% data per round, the surrogate distribution gap between global and local models (red line in Fig. 2(c)) exceeds that observed with 50% data participation (pink line). This insight prompts global perspective by leveraging statistical information that partially reflects the data distribution. Unlike prior approaches that focus on single aspect, our method coordinates both distribution and parameter spaces, enhancing robustness across diverse FL heterogeneous scenarios.
investigate the parameter space, where model updates gain more
4.2 Static Goal-oriented Objective
Building on the motivation outlined above, we propose a static goal-oriented objective function that changes each client's 's learning goal to better generalize on the global distribution D, by two-stage alignment (as depicted in Fig. 3(a)), rather than solely optimizing for the local distribution Dz. Firstly, we give the formal definition of the surrogate dataset in Definition 4.1.
Definition 4.1 (Surrogate Dataset). FedGPS assigns a distinct Gaussian distribution to each class in the original dataset. The surrogate dataset D then generated by sampling from these class-specific Gaussian distributions, with each client holding the same surrogate dataset D*
Then, we decompose the model parameters into a classifier and a feature extractor &b (where 0 = h 4h) cause we perform the alignment in the feature space. Ps: represents the probability distribution of the local data in the feature space, induced by applying the feature extractor th, to samples Êx. where ÉK Dr- Following, Pg is the distribution of k-th local surrogate distribution with D* and the global surrogate distribution Pa is aggregated at the server side. Specifically, we give the formal definition of local surrogate distribution and global surrogate distribution follows:
Definition 4.2 (Local Surrogate Distribution). For client k in federated learning system, the local surrogate distribution Pž is conceptually defined as the set of feature embeddings obtained by passing each data point from the surrogate dataset through the k-th local model's feature extractor sb at the client side.
In the implementation, ensure privacy and reduce communication overhead, what is transmitted to the server is compressed, mriacc-preeerving statistical representation of the surrogate distribution. Typically, these embeddings then aggregated Ex Te ΣE--D (bx(E() form a set of class-wise prototype vectors (e.g., 512-dimensional), with each prototype representing a specific class This distribution serves as compact proxy for the local surrogate distribution.
5

---

Definition 4.3 (Global Surrogate Distribution). At the server side, the global surrogate distribution Pp' is defined as the set feature embeddings obtained by passing each data point from the surrogate dataset through the global extractor ap.
To alleviate the burden of the global surrogate distribution compute, the global surrogate distribution is replaced with the aggregation of selected local surrogate prototypes E = Exes, E at round t. Furthermore, introduce the following theorem to formalize the new objective and quantify the aligmment between the local and global distributions, which establishes bounds on the Wasserstein- 1 distance between the distribution gap we analyzed before.
Theorem 4.4. Given the global feature distribution Pg. the local feature distribution Pk. the surrogate distributions " (global) and Pt (local for the k-th client) over their corresponding data space. Suppose there exists K. 2 O such that (PE,D. Pg.D) S K. under distribution D. If the following conditions hold:
(P Pk) Sc where W1 is the Wasserstein-I distance as defined in Definition 3.1. then the Wasserstein-1 distance between Pg and Pa is bounded as:
WI(PI: P*) <
+ W1 y: PT) +2 62 +.
Remark 1. This theorem establishes key relationship between local and global feature distributions. Specifically, suppose each client's local surrogate feature distribution P; closely approximates its true local feature distribution PA within tolerance of 61 (Stage f). Additionally, assume Pj aligns with the global surrogate feature distribution Dd within a tolerance of (2 (Stage /). Furthermore, let the local and global feature extractors produce similar outputs for identical data, within a tolerance of K. Under these conditions, the global model's feature distribution Pa will closely resemble p (Detailed proof can be found in the Appendix A).
The bound + 62 + ensures that a model trained on surrogate data generalizes effectively to the true global data, practice, 61 and 62 can be optimized using can be minimized through parameter regularization. Accordingly, our local optimization goal of each client can be formulated as follows:
losses, while
(Ox) ;= (EE=-,,E(0ss) Ex) + LEg,~--((O;; {.) + Aid(PR, Pf) + Agd(P", PPA) + 1x (6)
where the first two terms enhance the generalization of the local model 02 on both the local data distribution Dj and the surrogate data distribution D The function d(.,-) quantifies the distance between distributions, such as the Wasserstein-1 distance. The terms weighted by hyperparameters \1. Ag. and λ control the trade-off between terms, which are tuned to optimize performance.
4.3 Dynamic Path-oriented Rectification
To overcome the limitations of scarce data involved every round in achieving distribution alignment (demonstrated by Fig. 2(c)), we develop another technique, dynamic path-oriented gradient rectification, bolster model robustness. Our motivation draws high-level concept from the model replacement strategy in the backdoor of federated models [25]. In this scenario, the malicious client exploits a deep understanding of the aggregation mechanism and the collective dynamics of benign clients. By precisely predicting the contributions of other clients' updates to the global model, the attacker meticulously designs and scales their malicious update. The key insight that awareness of the aggregated influence from other clients confers substantial leverage in shaping the global model.
We define the gradient statistical information from other clients in Definition 4.5. Then we elaborate on how to utilize this information to improve the local update from a more global perspective at the gradient level (as depicted in Fig. 3(b)..
Definition 4.5 (Non-Self Gradient at Round t of client i, 5%,), In a Fr.aeewwokk with a client set K., let 0--1 denote the global model parameters at the end of round I - 1, and S.-1 C K. the subset of clients selected for round - 1 and |S,-1| 2 2. For each client l E K., Ab be the updated
information of client i round - 1, where A =0 -.-- and local update steps, respectively.
00-1 Let ny and ni denote the global
For client E K., the Non-Self Gradient at round of client i, denoted 8y,, is defined as:
5% = -1gg S
Σ 1/11
fiH

---

Table 1: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on CIFAR-10, 10, heterogeneity degree a 0.1, local epochs = and total client number K" 10.
Ditract Helanngneky Levalor
Number
Clism Sampling Boae:
Commanicann Rossi:
Local RipocheR
Heteogeteoos scenara
Hotoogeacoos 2ccoario
Heterggnscces scceatio
Hetorignaees soceario
Hetelggeeers socaurio
Cor Traising Aoowooad
ACCT ROUND y Spealtyy
ACC
ROUND Spoollip
AGC) BOUND 1 Spccllip
ACCC BOUND 1 Speedlpp
ACC1 ROUND, Speedlipi
Metbods
Target Acc-545
Taget Acc-7965
Tauget Acc-dDS
Tagget Acce685
Target Aoow6155
MMein Adee Std
Ped/og
64.21
240
1.3K
73.13
-333
L/KK
50.63
436
1,0x
56.022
189
1,0X
62.55
415
1.0x
75.00 = 7.99
FallagM
65.24
181
1.2%
51.25
2300
1.5%
61.55
510
1.3
70.15
515
0.3%
$7:51
233
1.5x
77.51 8 7.98
FodPecs
175
68.81
SCAFYOLD
61.30
391
0.9%
83.25
2.3%
13.30
222
76.57
55
986
24.12
88
633
80:41
VHL
80.07
116
2,8
57:30
131
2.3%
68.83
210
280
54.90
50
2.1%
51.06
100
288
55.00
3.10
PrdASAM
09:40
0770
1.3x
51
211
1.4%
10.15
310
13x
75:11
185
1.0%
06.00
588
1.2
77.78 4 7.81
Fallipp
64.00
230
1.3%
19.25
211
1.4%
10.00
None
Nenz
71.85
185
1.0%
$6.90
515
135
75.21 16.37
FolDeorm
-300
52.07
51.35
355
1.2%
73.17
70.02 8.585
Fedlnx
53.43
1.2X
50.43
180x
50.44
815
0.6K
170
77.00 2:44
FdLEESM
85.80
121
2.3x
55.02
1.20
2.3%
64.21
213
1,8x
73.191
90
2.1%
18
119
2.5%
82:00 5.77
NECFL
83.26
None
Noce
19:45
325
0.5%
70.08
Nore
Noes
58.
21.
0.5%
65.78
687
1.3%
75.61 #:277
90.31
139
2.4%
88 46
119
2.5%
87.78
158
2.6
85.00
80
2.1
52
04
187
8.0x
98.
98.79-2 9:28
Table 2: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on SVHN, heterogeneity degree a: 0.1, local epochs E 1 and total client number K 10.
Davoer SVHN Heanoppoitty Levalio
0.1 Clerr Namber K
10 Clisst Sumpling Rowe 50% Tixal Conmenanizatinn Roadd/
500
Eppacts
Soasuri
||
Heteoggaeces scerario
Heonogenenus scenarie
Heteogenocos scerorio
Heterggrnenns sonsurio
Heonogenoous sconario
Comnalted Training
NCD1 BOUND, i Speedipt NOCT ROUND Spedtipt AEC! ROUND I Speetl yr NCCT BOUNND Speedkipt NOC ROUND Spoottig)
Methed
||
Digget Acced55i
Turgot Acco0224
Tareet Acc
Turget Accn0155
Target Acce
Mcas
Auci
Pedlor
55.01
15:
1.0
12.36
112
1.0%
12:15
1000
1.0%
$2.05
5:00
1.0%
52.11
65
1.0%
91.03
2/12
PedAugM
58.61
150
1.40x
92:41
110
0.9%
17231
141
1.5%
02 331
1:1
3.4%
13.33
61
1.0%
01.0
1.80
Folthas
58.03
102
1.5x
93.00
107
1.0x
10:15
154
52.50
104
3.3x
97.90
61
1.0%x
00.07
SCAFFOLD
67.88
93.58
Noat
1G
51
91.74
Noot
Noxe
No1t
1.6%
02.65
0.7x
VHL
58.87
94.50
0.0%
94.99
93
Lix
54.00
55
4.0
91.30
1.0%
01.05
0.61
Psd1SAM
58.14
150
10xx
78.56
107
1.0%
02.22
02
1.1
52.65
116
2.5
51.30
64
1.0K
91.55
2: 2:30
Telliop
56:21
150
18x
12.11
110
6.5%
91.57
Nons
None
52.00
2000
1.0
$2.85
64
1.0%
01.02
2:10
FoiDoomm
80.82
50
19x
12.50
255
0.4
50.02
71
1.4%
58.19
182
190
91.11
04
1.0%
52.43
1.00
FetDacc
$4.51
None:
Nox
92.90
1.x
90.30
10
1.0%
91.91
Nave
Nonz
97.85
1.0%
04.99
3.58
Ninz
50:29
93.022
0.64
1.6X
9.5%
91.71
1.0%
14.127
NUCIT
96.49
118
1.3
90.53
Nove
Non
01:03
Nove
Newo
91 36
Nare
Nom
91:92
Vier
Nom
RddCPS/Dus)
84:20
50
2.3%
55.22
07
1.6%
05.29
49
2.0%
05.23
73
4.7%
196.05
39
1.7%
06.00 0.45
where S \ {2} is the set of clients in S excluding client i, and [S \ {i) is its cardinality.
By integrating this definition, the local client concurrently considers non-self gradient information before computing the update direction, as this information subtly conveys the underlying data distribution from others, which can be expressed as:
++1,+1 8k
V PP + Xg
%
Here, represents the e-th local update iteration within a total of E local epochs per round. The 5b
56, denotes unit vector aligned with the direction of 6%. We employ the Xg To, T
expression
instead of 5%. to focus exclusively on the update direction from other clients, with the hyperparameter Ag providing adjustable scaling to optimize performance. Lastly, the local model 0 gf+1. updated by the new rectified path as follows:
(8) This term is deemed dynamic as the gradient path is adjusted at each update iteration. The local update direction is consistently refined using statistical gradient information from other clients.
0 +L+++1 = 0 +1. migk +1,++1
5 Experiments
We organize this section as follows: (a) Detailed description of all the evaluated methods in our comprehensive evaluation (Sec 5.1); (b) The implementation and experimental settings we followed (Sec 5.2); (c) The main results and observations to demonstrate the efficacy of FedGPS (Sec 5.3); (d) Ablation study on two modules of FedGPS (Sec. 5.4).
7

---

Table 3: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on CIFAR- 100, heterogeneity degree a 0.1, local epochs E and total client number K 10.
Dipad:
CIFAB-10O Beunogsnelly
Number
Sumplny 50% Total Communattion Bound
Epache
Scoserio
Hetoogeteoos scenaro
Hetoogeacoos Sccoario Heterogescces 3cceatio
Hetorogeabiss s
Hetelggeeers socaario
Cestrllied Traniy AcceTee
ACCT ROUND j Spealty
ACC
ROUND Spedltip
ACE) BOUND y Specdlip ACCC
BOUND y Speedlip ACC1 ROUND, Syeedlipi
Metbods
Target Accedtrs
Tager Acceddts
Tagget AccH6995
Target Aco-70%
Target Nocwlols
MMein Adce: Std
Peddog
00.29
200
1.3k
10:05
413
L/K
(0.13
-7)
1,0X
70.02
429
1,0
86.04
435
1.0x
68,00 = L.44
Fallogh
10.10
350
00.14
476
0.3
60.00
000
1.2
10.02
634
1.0x
85
691
0.9
69.52 a 1.44
FolPecs
460
67.45
Nore
Note
SCAFYOLD
VHL
20.05
824
1.5x
00.35
407
1/8x
70.05
601
1.2x
21.03
015
1.1%
65.77
500
14c
70.16
Prd4SAM
70.01
350
1.1%
08:35
Now
Neve:
10:337
34P
1.2g
70:71
131
1.k
06.52
175
1.0%
69:11 1: 1.40
Follicp
60:72
-115
1.2%
08.00
476
0.3%
00.01
C25
1.1
20.03
153
1.0c
05.51
Nonz
Narz
51 1.2.2
FolDeorm
(2:93
08:85
Nore
No00
655
02.00
2:331
Fedlnx
07.87
None
00.02
None
Nose
Note
89.41
Note
Muse
63.50
Nose
00.01
FeLEESM
08:24
Kone
Nove
67:33
Nove
Nove
66.57
Noce
No0e
57.82
Note
Noce
62.61
Nore
Note
67. 07:28-0
1:23
XUCFL
16.29
None
Newe
67.94
Neve
Non:
66.47
Nore
Noxe
51.81
Noxe
Nose
54.64
Nose
Nosc
05.70
1.72
FourPS:Our)
71:14
838
1,5x
70.58
427
1.0%
70.50
374
1.3
71.44
400
1.1%
09.79
203
15x
70:09 4 0.64
5.1 Evaluated Details
We evaluate the FL methods that alleviate data heterogeneity from different perspectives. 1) FedAvg [7] is the fundamental work in FL; 2) FedAvgM [51] accumulate model updates with momentum; 3) FedProx [11] constrain the divergence between local and global models; 4) SCAFFOLD [12] use extra term to correct the local gradients: 5) CCVR [13] share statistical logits to sample rectification data at the server side; 6) VHL [14] use virtual homogeneity data constrain model by domain adaptation. 7) FedASAM [26] and Fed FedLESAMS [29] use the insight of sharpness aware minimization; 8) FedExp [52] is inspired by Projection Onto Convex Sets (POCS) to select global step size adaptively; 9) FedDecorr [53, 54] constrain the feature covariance matrix due to the dimension collapse; 10) FedDisco [21] adjusts the aggregation weight based on discrepancy between clients; 11) FedInit [55] improves the local consistency by related initialization; 12) NUCFL [56] calibrates local classifier after local training.
Compared Methods:
Datasets, Models and Metries: Following [3. 14, 57], we evaluate our method on three standard datasets: CIFAR-10, CIFAR- 100 [58]. and SVHN [59]. In line with prior work [57, 60], we use ResNet-18 for CIFAR- 10 and SVHN, and ResNet-50 for CIFAR-100. We report three metrics related to communication efficiency and performance, building on previous work [14, 15]: (1) "ACC": the best accuracy achieved during training, with the target accuracy set as the best performance of FedAvg to provide a lower bound for evaluation; (2) "ROUND": the communication round required to reach the target accuracy; and (3) "SpeedUp"": the speedup factor compared to FedAvg.
5.2 Implementation Details
To simulate a heterogeneous data distribution across clients, we employ the Dirichlet partitioning method, a common approach in recent FL works [8, 51J. This method draws client data proportions q from Dirichlet distribution, q Dir(ap), where is the concentration parameter that controls the degree of heterogeneity. We use a = 0.1, but vary the random seed to generate multiple distinct heterogencous data distributions. Examples of these distributions shown in Fig. 2(a) and 2(b). We simulate cross-silo scenarios using 10 clients and cross-device scenarios using 100 clients. We set the sampling rate A, as 50% for cross-silos and 10% for cross- devices scenario. We set local epochs E' = 1 (results for different local epochs are shown in the Appendix D.3).
Federated Settings:
To ensure a fair and direct comparison, all methods were evaluated under identical conditions, including the same data partitioning, sampling rate, local epochs, and communi- cation rounds. We use the SGD optimizer with 0.01 learning rate and 0.9 momentum, le-5 weight decay (also denoted as Ag). Among the hyperparameters, ê and X2 were both set to 0.1, and Ag is experimeet experiments (Details can be seen in the Appendix C).
Experimental Details:
fixed at 0.5 for the main
5.3 Main Results
Our evaluation results on CI, SVHN, and CIFAR-100 are shown respectively in Tabs. 1, 2, 3 and 4. Additionally, if a method fails to produce valid results, e.g., NaN loss, we denote its
8

---

Table 4: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on CIFAR-10, 10, heterogeneity degree a 0.1, local epochs and total client number 100.
Hetengenccyy
Numben
Sempling
Commaniaiion Round
Epocha
Hetoogeteoos scenara
Hetoogeaccos 2ccoario
Heterggnnoces scceatio
Helonggaaees succatio
Heielggeeers socsario 5
Cor Traising Aooeocad
ACCT ROUND j Speally
ACC
ROUND Spoolipp
ACC) BOUND 1 Specll.p
ACC) BOUND 1 Speedupp
ACC1 ROUND, Speedlipi
Metbods
Target Acc-488
Taget Acce4555
Tauget Acc-571
Target Acce355
Target Acte3955
Meei Adce Std
Ped/og
05.22
449
1.3K
68.23
412
1,Kx
27.61
1,8x
30.15
479
1,0X
35:41
4015
1.0x
45.00
7.45
Fallggh4
19.80
305
1.5%
00.07
355
1.2
68.00)
<14
1.2
66.80
201
1.8x
15.21
652
1.2x
55.45
9,09
FodPecs
64.15
570
SCAFYOLD
61.08
68.03
5.3%
201
1.2
38
12.8x
91.13
01.00
VHL
72.20
128
3.5%
10.21
201
2.2%
16.12
235
2,1
18.18
145
9.3%
52.64
123
5.3%
00.00
6.15
PrdASAM
06.35
Nevs
Noms
15:33
Non
Nom
53:35
Nor
Nevn
11.50
075
1.0%
33.07
Nrnt
Nar
11:23
7:35
Fallepp
56.26
Nenz
Nams
16.25
Nor
Norc:
16.01
None
Nonz
15.53
367
133x
37.55
Norz
Nazz
11.50
7.25
FoDDeom
.003
05.55
3317
1.4%
Fedinx
72.09
1.4x
00:54
FdLEESM
72.04
130
4.3x
76.45
1,46
3.3x
75.47
211
2.1x
77.06
75
6.4%
73.73
70
6.5%
74.06 =:
XUCFL
63.72
325
1.4%
02.85
207
1.5%
18.80
Nore
Noes
49.67
231
2.1%
45.17
500
144%
$1.20 3.33
FedCPS/Ours)
78
102
4.4%
70.97
155
28%
70.27
232
2.3%
78.12
94
5.1X
75.08
70
6.0%
77 77.08
1.19
15
Fabito
100%
Ieperseent
XUPG
300
(a) Ablation results different part of FedGPS.
(b) Test accuracy and convergence rate on different baselines and FedGPS.
Figure 4: The visualization of the ablation study and convergence of FedGPS compared with other baselines. Due the large volume of baselines, we select the top 5 baselines plot.
performance "-" our results. Based on our experiments, we outline several key observations that highlight the characteristics of the evaluated methods and provide insights for future research:
Observation 1: Absence of SOTA Methods Across Scenarios. The results indicate that most methods exhibit limited robustness across various scenarios. For instance, VHL demonstrates superior performance compared to other methods under the setting a 0.1, K 10, A, 50%. However, on the same dataset with a = 0.1, K = 100, A = 10% its performance degrades significantly. Similar patterns are observed in other methods, suggesting that the performance of given method can vary substantially across different settings.
Observation 2: Value of global classifier calibration. Global classifier calibration proves to be effective in certain contexts. For example, CCVR employs logits-based statistical information sampled from Gaussian distribution to calibrate the classifier (h) globally. This approach reduces the number of communication rounds required to achieve the target accuracy on specific datasets, also stabilizes the training curve. As shown in Tab. 4, under certain distributions, CCVR achieves the target accuracy with fewer communication rounds compared to our method, despite lower overall performance. This observation inspires future research to enhance performance using such techniques.
Observation 3: Performance variability across settings. The performance of methods varies
significantly across different settings, indicating need for improved adaptability meticulous hyperparameter tuning. For instance, FedASAM and FedExp outperform vanilla FedAvg under 0.1, 10, X 50%, but struggle to surpass FedAvg under a 0.1, K 100, X 10%. Similarly, many methods achieve performance comparable to or worse than FedAvg on CIFAR-100, underscoring the challenge of generalizing across diverse datasets and configurations.
Our experimental results demonstrate that FedGPS consistently achieves state-of-the-an (SOTA) per- formance across diverse federated learning settings and datasets. As reported in Tab. 1, FedGPS sur- passes the best baseline methods under various heterogeneous scenarios. However, its performance gains on SVHN are modest, vanilla FedAvg already approximates centralized training perfor-

---

mance in these scenarios, limiting the potential for improvement in distributed settings. In more challenging environments, such as those detailed in Tab. 3, FedGPS exhibits substantially greater improvements. Crucially, FedGPS prioritizes robustness across heterogeneous data partitions over optimizing for specific scenarios' performance, design choice that enhances its generalization ability. The convergence rates of different evaluated methods and FedGPS are shown in Fig. 4(b).
5.4 Ablation Study on Two Perspectives
To evaluate the individual contributions of the static goal-oriented objective function and the dynamic path-oriented gradient rectification in FedGPS to FL performance, we conduct ablation studies by equipping FedAvg with each module in isolation. As shown in Fig. 4(a), the results report relative per- formance improvements over the target accuracy of vanilla FedAvg. Specifically, FedGPS without the static objective modification relies exclusively on dynamic path rectification, whereas FedGPS with- out dynamic path rectification employs only the static objective function. These experiments confirm the distinct effectiveness of each module. Notably, the synergistic integration of both modules yields superior performance across diverse heterogeneous scenarios. Additional ablation studies, exploring varying numbers of clients, datasets, and local epochs, are detailed in the Appendix D. Furthermore, to assess the robustness of FedGPS under different training seeds, we initialize the model with three different random seeds under identical settings and data distribution. Its comprehensive results are provided in the experimental section of the Appendix D.8.
Conclusion and Further Discussion
In this work, we explore an important and overlooked question: how well do existing notable algorithms perform in multiple heterogeneous scenarios? Extensive experiments show that most of the existing algorithms are limited robustness, which inspired the FedGPS framework. combines two orthogonal views to achieve abbel-disribbution-agoois information the client from the distribution level and the gradient view, respectively. More analysis about the communication and privacy of FedGPS are listed in the Appendix B. It also catalyzes future research into datribution-agnoste algorithms, paving the way for resilient federated learning in complex, real-world settings.
robustness by considering the statistical
Limitations: Limited computational resources may constrain the performance of FedGPS, as FedGPS relies on additional surrogate data for its distribution alignment process. Future work could investigate more efficient alignment techniques that minimize the need for surrogate data or explore alternative approaches to enhance scalability. Furthermore, FedGPS does not yet address challenges posed by heterogeneous data features, necessitating further research into the robustness of its distribution and gradient collaboration framework across a broader range of heterogeneous FL scenarios, with the goal of achieving distributionnaanostii robustness.
Broader impacts
Federated learning (FL), defined by its distributed data collection and keeping data locally, inherently navigates complex real-world applications driven by diverse tasks and participants. This complexity has spurred extensive exploration of varied federated settings. Our work tackles critical challenge in FL: the pervasive data heterogeneity that undermines the robustness of existing methods across diverse data distributions. Through over 1100+ groups of experiments, we investigate mitigation strategies from multiple perspectives, introducing novel insights that significantly enhance robustness. We also provide key observations guide future research and inform the selection of federated methods for heterogeneous scenarios. Rather than advocating for a single algorithm tailored to a specific scenario, we emphasize the need for broader, actionable insights to support practical FL deployments, enabling customized solutions for diverse applications. This paper marks a pivotal step toward distribution-agnosti federated learning, establishing a foundation for robust, scalable, and adaptable FL. systems. By bridging experimentation with practical applicability. our contributions aim to catalyze transformative advancements in this rapidly evolving field and future real applications with sensitive data protection.
10

---

Acknowledgements
We thank all the reviewers for their constructive suggestions and dedication to this paper. ZQY, CXL, and YXY were supported by Hong Kong Innovation and Technology Commission Innovation and Technology Fund ITSS/229/2.. YGZ was funded by Inno HK Generative AI R&D Center. BH was supported by NSFC General Program No. 62376235 and RGC General Research Fund No. 12200725. YMC was supported by the Hong Kong Baptist University (HKBU) under grant RC-FNRA-IG/23- 24/SC1/02, and the seed funding for collaborative research grants RC-SFCRRG/33-4RR2/SCII
References
[1] Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
[2] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2): 1-000, 2021.
[3] Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, Hi, Bo Du, and Qiang Yang. Federated learning for generalization, robustness, fairness: A survey and benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[4] Shaoxiong Ji, Yue Tan, Teemu Saravirta, Zhiqin Yang, Yixin Liu, Lauri Vasankari, Shirui Pan, Guodong Long, and Anwar Walid. Emerging trends in federated learning: From model fusion to federated x learning. International Journal of Machine Learning and Cyberneties, 15(9):3769-3790 2024.
[5] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE transactions on neural networks and learning systems, 34(12)95587-603, 2022.
[6] Benyuan Sun, Hongxing Huo, Yi Yang, and Bo Bai. Partialfed: Cross-domain personalized federated learning via partial initialization, Advances in Neural Information Processing Systems, 34:23309-23320, 2021.
[7] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efiin -fffccin learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[8] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th international conference on data engineering (ICDE), pages 965-978. IEEE, 2022.
[9] Ming Hu, Yue Cao, Anran Li, Zhiming Li, Chengwei Liu, Tianlin Li, Mingsong Chen, and Yang Liu. Fedmut: Generalized federated learning via stochastic mutation. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 12528-12537, 2024.
[10] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE signal processing magazine, 37(3):00-60, 2020.
[11] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
[12] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132-5143. PMLR, 2020.
[13] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of hetero- geneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984, 2021.

---

[14] Zhenheng Tang. Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Vir- tual homogeneity learning: Defending against data heterogeneity federated learning. In International Conference on Machine Learning, pages 21111-21132. PMLR, 2022.
[15] Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. Fedfed: Feature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36, 2023.
[16] Lei Shen, Zhenheng Tang, Lijun Wu, Yonggang Zhang, Xiaowen Chu, Tao Qin, and Bo Han. Hot-pluggable federated learning: Bridging general and personalized fl via dynamic selection. In The Thirteenth International Conference on Learning Representations, 2025.
[17] Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:761 2020.
[18] Wenjing Yan, Kai Zhang, Xiaolu Wang, and Xuanyu Cao. cabblem-parameterr federated learning. In The Thirteenth International Conference on Learning Representations, 2025.
[19] Wenlin Chen, Samuel Horvath, and Peter Richtarik. Optimal client sampling for federated learning. Transactions on Machine Learning Research, 2022.
[20] Qing Li, Shanxiang Lyu, and Jinming Wen. Optimal client selection of federated learning based on compressed sensing. IEEE Transactions on Information Forensics and Security, 2025.
[21] Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang. Feddisco: Federated learning with diserepancy-aware collaboration. In International Conference on Machine Learning, pages 19879-39902. PMLR, 2023.
[22] Jiahao Liu, Jiang Wu, Jinyu Chen, Miao Hu, Yipeng Zhou, and Di Wu. Feddwa: personalized federated learning with dynamic weight adjustment. In Proceedings of the Thirty-Second International Joint Conference Artificial Intelligence, pages 3993 4001, 2023.
[23] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning glohal model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10174-10183, 2022.
[24] Yifei He, Haoxiang Wang. Bo Li, and Han Zhao Gradual domain adaptation: Theory and algorithms. Journal of Machine Learning Research, 25(361):1 2024.
[25] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International conference on artificial intelligence and statistics, pages 2938-2948. PMLR, 2020.
[26] Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In European Conference on Computer Vision, pages 654-672. Springer, 2022.
[27] Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International conference on machine learning, pages 18250-18280 PMLR, 2022.
(28) Rong Dai, Xun Yang, Yan Sun, Li Shen, Xinmei Tian, Meng Wang, and Yongdong Zhang. Fedgamma: Federated learning with global sharpness-awaan minimization. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[29] Ziqing Fan, Shengehao Hu, Jiangchao Yao, Gang Niu, Ya Zhang, Masashi Sugiyama, and Yanfeng Wang. Locally estimated global perturbations are better than local perturbations for federated sharpness-aware minimization. In Forty-first International Conference on Machine Learning, 2024.
[30] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
12

---

[31] Yue Tan, Guodong Long, Lu Liu,, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4432-8440, 2022.
[32] Xin Gao, Xin Yang. Hao Yu, Yan Kang, and Tianrui Li. Fedprok: Trustworthy federated class-incrementa learning via prototypical feature knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4205 42054421, 2024.
[33] Haiyang Guo, Fei Zhu, Wenzhuo Liu, Xu- Yao Zhang, and Cheng- Liu. Pilora: Prototype guided incremental lora for federated dlass-incrementae learning. In European Conference on Computer Vision, pages 141-159. Springer, 2025.
[34] Chuanneng Sun, Tingcong Jiang, and Dario Pompili. Heterogeneous federated learning via generative model-aided knowledge distillation in the edge. IEEE Internet of Things Journal, 2024.
[35] Jinglin Liang, Jin Zhong, Hanlin Gu, Zhongqi Lu, Xingxing Tang, Gang Dai, Shuangping Huang, Lixin Fan, and Qiang Yang, Diffusion-driven data replay: novel approach to combat forgetting in federated class continual learning. In European Conference on Computer Vision, pages 303-319. Springer, 2025.
[36] Yunpeng Zhao and Jie Zhang. Does training with synthetic data truly protect privacy? In The Thirteenth International Conference on Learning Representations, 2025.
[37] Wenlin Chen, Samuel Horváth, and Peter Richtárik, Optimal client sampling for federated learning. Transactions on Machine Learning Research, 2022.
[38] Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In IEEE INFOCOM 2022-IEEE conference on computer communications, pages 1739-1748. IEEE, 2022.
[39] Yann Fraboni, Richard Vidal, Lactitia Kameni, and Marco Lorenzi. Clustered sampling: Low-variance and improved representativity for clients selection in federated learning. In International Conference on Machine Learning, pages 3407-3416 PMLR, 2021.
[40] Zexi Li, Tao Lin, Xinyi Shang, and Chao Wu. Revisiting weighted aggregation in federated learning with neural networks. In International Conference on Machine Learning, pages 19767-19788. PMLR, 2023.
[41] Ming Hu, Zhihao Yue, Xiaofei Xie, Cheng Chen, Yihao Huang, Xian Wei, Xiang Lian, Yang Liu, and Mingsong Chen. Is aggregation the only choice? federated learning via layer-wise model recombination. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1096 1107, 2024.
[42] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heteroge- neous federated learning. In International conference on machine learning, pages 12878-12889, PMLR, 2021.
[43] Yuan Wang, Huazhu Fu, Renuga Kanagavelu, Qingsong Wei, Yong Liu, and Rick Siow Mong Goh. An aggregation-free federated learning for tackling data heterogeneity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26233-26242,
2024.
[44] Yifan Zhang, Ying Wei, Qingyao Wu, Peilin Zhao, Shuaicheng Niu, Junzhou Huang, and Mingkui Tan. Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Transactions on Image Processing, 29:7834-7844, 2020.
[45] Shengyuan Liu, Ruofan Zhang, Mengjie Fang, Hailin Li, Tianwang Xun, Zipei Wang, Wenting Shang, Jie Tian, and Di Dong. Perfed: personalized federated learning with contrastive representation for non-independently and identically distributed medical image segmentation. Visual Computing for Industry, Biomedicine, and Art, 8(1):6, 2025.
13

---

[46] Linlin You, Rui Zhu, Mei-Po Kwan, Min Chen, Fan Zhang, Bisheng Yang, Man Sing Wong, and Zheng Qin. Unraveling adaptive changes in electric vehicle charging behavior toward the postpandemic era by federated meta-learning. The Innovation, 5(2), 2024.
[47] Yonggang Zhang, Zhiqin Yang, Xinmei Tian, Nannan Wang, Tongliang Liu, and Bo Han. Robust training of federated models with extremely label deficiency. arXiv preprint arXiv:2402. 14430, 2024.
[48] Zhuoran Ma, Yang Liu, Yinbin Miao, Guowen Xu, Ximeng Liu, Jianfeng Ma, and Robert H Deng, Flgan: Gan-based unbiased federated learning under non-iid settings. IEEE Transactions on Knowledge and Data Engineering. 36(4):1566-158) 2023.
[49] Jiezhang Cao, Yong Guo, Qingyao Wu, Chunhua Shen, Junzhou Huang, and Mingkui Tan. Improving generative adversarial networks with local coordinate coding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):211-221) 2022.
[50] Mingkui Tan, Peihao Chen, Hongyan Zhi, Jiajie Mai, Benjamin Rosman, Dongyu Ji, and Runhao Zeng. Source-free elastic model adaptation for vision-anddlanguuge navigation. IEEE Transactions on Mullimedia, pages 1-13, 2025.
[51] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint vrXiv:1909.000 06335, 2019.
[52] Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated aver- aging via extrapolation. In The Eleventh International Conference on Learning Representations, 2023.
[53] Yujun Shi, Jian Liang, Wenqing Zhang, Chuhui Xue, Vineliu202Spefeedett YF Tan, and Song Bai. Understanding and mitigating dimensional collapse in federated learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[54] Yujun Shi, Jian Liang, Wenqing Zhang, Vincent Tan, and Song Bai. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. The Eleventh International Conference on Learning Representations, 2023.
[55] Yan Sun, Li Shen, and Dacheng Tao. Understanding how consistency works in federated learning via stage-wise relaxed initialization. Advances in Neural Information Processing Systems, 04880043-805747 2023.
[56] Yun-Wei Chu, Dong-Jun Han, Seyyedali Hosseinalipour, and Christopher Brinton. Unlocking the potential of model calibration in federated learning. In The Thirteenth International Conference on Learning Representations, 2025.
[57] Qinbin Li, Bingsheng He, and Dawn Song. of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713-10722 2021.
federated learning. In Proceedings
[58] A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[59] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, page 4. Granada, 201
[60] Sara Pieri, Jose Restom, Samuel Horvath, and Hisham Cholakkal. Handling data heterogeneity via architectural design for federated visual recognition. Advances in Neural Information Processing Systems, 36:4115-4136 2023.
[61] Lei Wang, Jieming Bian, Letian Zhang, Chen Chen, and Jie Xu. Taming cross-domain represen- tation variance in federated prototype learning with heterogeneous data domains. Advances in Neural Information Processing Systems, 08883388883722 2024.
[62] Peter Bjom Nemenyi. Distributtin-free multiple comparisons. Princeton University, 1963.
14

---

[63] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan): 1-30, 2006.
[64] Xiangzhi Chen, Le Wu, Fei Liu, Lei Chen, Kun Zhang, Richang Hong, and Meng Wang. Disentangling cognitive diagnosis with limited exercise labels. Advances in Neural Information Processing Systems, 36: 36:18008-18888 -0004, 2023.
[65] Christoph Jansen, Georg Schollmeyer, Julian Rodemann, Hannah Blocher, and Thomas Au- gustin. Statistical multicriteria benchmarking via the gsd-front. Advances in Neural Information Processing Systems, 11-988143-11777 2024.
[66] Yuwen Yang, Yuxiang Lu, Suizhi Huang, Shalayiding Sirejiding, Hongtao Lu, and Yue Ding. Federated multi-task learning on non-iid data silos: An experimental study. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 684 693, 2024.
[67] Yukun Song, Dayuan Cao, Jiali Miao, Shuai Yang. and Kui Yu. Causal multi-label feature selection in federated setting. arXiv preprint arXiv:2403.000 0224403004999 19, 2024.
[68] Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021.
[69] Junhyung Lyle Kim, Taha Toghani, Cesar A Uribe, and Anastasios Kyrillidis. Adaptive federated learning with auto-tuned clients. In The Twelfth International Conference Learning Representations, 2024.
[70] Sohom Mukherjee, Nicolas Loizou, and Sebastian U Stich. Locally adaptive federated learning. Transactions on Machine Learning Research, 2024.
[71] Ming Hu, Peiheng Zhou, Zhihao Yue, Zhiwei Ling, Yihao Huang, Anran Li, Yang Liu, Xiang Lian, and Mingsong Chen. Fedcross: Towards accurate federated learning via multi-model cross-aggregation. 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 2137-2150. IEEE, 2024.
[72] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint /rXiv:/866. 880.00088. 2018.
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We list our contributions in the introduction part.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of this work in the conclusion part.
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
Due to the limited page, we include this section in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We list all the experimental hyperparameters in the main and supplemental material each settings, and the hyperparameters of baselines are also listed.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data are open access, and we will open- source our code when this paper gets accepted. All the hyperparameters in our experiments are listed in main paper and supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Saliiiigguidss/Codeeee wttsssubmisiiooooooooo.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cttpccppp/....... ccc/pbllc//aaeeeccccccccccececce hwt.c//gggee//eeeeeeeeeee /Cee/oooouummiic wwt.SSubmissiii ssssnnolliii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We list these information in the experimental part and supplemental material.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We use some statistical significance method to evaluate both baselines and other method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is listed in the section 5.2 of experimental details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //wurri///./.s.eeeeeee cc/ wwtpcccccpccceeeeeeeeece petubli//tthicoo wwwhiiihiieeeeeeeeeeeeeeeeeeeeee
Answer:[Yer [Ye]]
Justification: We conduct this paper under the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include this discussion in the appendix due to the limited pages.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Not Applicable in this work.
Guidelines:
The answer NA means that the paper poses such risks.
20

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite the relevant papers in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thoode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not Applicable in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not Applicable in this work.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not Applicable in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Not Applicable in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /ttpe:///eeup.peeeeeeeee http://ewiip.eceeeeeeeeeeeeeeeee www.c///eoeeeeee//////////a for what should or should not be described.
22

---

Appendix
Proof Results
24
A.1 Proof of Necessary Lemma A.3
24
A.2 Proof of Necessary Lemma A.4
25
A.3 Proof of Theorem 4.1
25
More Facts about FedGPS
26
B.1
Communication Analysis of FedGPS
27
B.2 Privacy Analysis of FedGPS
28
B.3 Brief introduction of Nemenyi post-hoc test method
28
B.4
Theoretical Justification of Dynamic Path-oriented Rectification
28
More Experimental Details
29
C.1 More Data Distribution
29
C.2 Detailed Hyperparameters
31
C,3 Process and Pseudocode of Algorithm
D Further Experimental Results
32
D.I More Baselines
33
D.2 Ablation Study on Client Number
................................
D.3 Ablation Study on Local Epoch E
34
D.4 Ablation Study Client Sampling Rate A
35
D.5 Ablation Study on Heterogeneity Degree a
37
D.6 Different Heterogeneity Partition Strategy
37
D.7 More Visualization of Results
37
D.8 Different Random Training Seeds
38
23

---

A Proof Results
Assumption A.1 (Lipschitz Continuity), For local feature extractor f :: X 2 parameterized by ab is I-Liipchitz continuous, that is,
(1 - Ju (x)llz S Lilbbx w/lllp where k E S
for all X E X. where w, x// E Blll Moreover || parameter spaces, respectively.
|lz and || 11 are norms on the feature and
Remark 2. This assumption ensures that small changes in the parameters of the local feature extractor lead to proportionally small changes the extracted features. Specifically, for any client k, the parameters sb are slightly modified to rp., the resulting feature representations remain close in the feature space Z, with the difference bounded by the Lipschitz constant L.
Lemma A.2. For random vectors V1, V2, Vns we have
T
Ival2a
t=1
1=1
A.1 Proof of Necessary Lemma A.3
Lemma A.3 (Bounded difference between global and local feature extractor sb. 1by). In FL with K clients, where each round t samples subset of clients St S S, and the global feature extractor parameters are updated as up'+1 [S.]] Ekes, bbx t+1 with local parameters ubk updated via bounded optimization, there exists As such that:
1/ 1112 Ad.
for all k E S, where - is the Euclidean norm on the parameter space.
Proof. Local parameters Ry; are updated using optimization (e.g., SGD) with regularization or gradient clipping, ensuring bounded norms. At round t, the global parameters are:
2/
1+1
S
KES.
Consider the parameter difference for any client k S, necessarily in St:
1/bb 2b(+1112
1+1
e+1
(tbx 2+1 - 2b 1+1)
|S
SSp
jES,
jES.
2b
1+1
l|2
(b)
||
- - +b, (+1 1|2
St
S
JES,
JES,
where (a) is from Lemma A.2 and (b) is because
||av||2 aalvvl2
Assume optimization bounds the parameter norm:
|/ 1+11/2 < B, for some B > 0, across all rounds
and clients (achieved via regularization). Then:
| ++1
"b
++1 112
||
++1
1|2 + 1| 5+1 112 < 2B.
Thus:
1 1+1
2B 2B.
S jES.
This bound holds for all & S, the maximum difference is independent of whether E S.. Set A 4 2B, so:
1 2 Ad.
which completes the proof of Lemma. A.3.
24

---

A.2 Proof of Necessary Lemma A.4
Lemma A... In FL with K clients, let the global feature extractor dp have parameters ub*+1 Till Ekes, 4hx. t+1 where abs. are parameters of local feature extractors. Let Pv.D and ggD denote the feature distributions induced by 4/y and up on a certain data distribution D. Given Assumption A.I and Lemma A.3, there exists LAdd, such that:
W.(P..DPPP.D) S K.
where W1 is the Wasserstein-I distance in the feature space 2.
Proof. The Wasserstein-1 distance is:
/ || z' (2122))(222))))
inf Wi Wi(Pk,D, Pg.D) DEII(E,DDP....
For r ~ D, by Assumption A.I, with f : X + Z parameterized by local feature extractor Thy and global feature extractor w respectively:
[x))(x) (x)x)l = [| 2h +1 مx<z; + || S Lllbbk (+1 *b*+111. 112-
By Lemma A.3, ///b2 1+1 abpt+112 1/2 I Aa. Thus:
||>b(<r) (x))l S LDs.
Define coupling 7 where 24: (x)) (bxx,, with probability D(x) and marginals:
First: S2 ((2,, 2) = D(x (x())) = 2ạ) = PR.D.
Second: Jg 22, D(x (xx) 2) Pg.D.
The cost is:
/ || - =||2 2 S LDd.
Thus:
(P S n, k LAd.
which completes the proof of Lemma A.4.
A.3 Proof of Theorem 4.1
Theorem 4.1. Given the global feature distribution Pg. the local feature distribution Ps. the surrogate distributions Ds (global) and Pi (local for the k-th client) over their corresponding data space. Suppose there exists x 2 0 such that Wi (PRP,,D Po.PP..0 Pg.D) S K under distribution D. If the following conditions hold:
El, where Wi is the Wasserstein-1 distance as defined in Definition 3.1. then the Wasserstein-1 distance between Pg and ps is bounded as:
(1 Px) S1,
Wi(Pi P*) S E2,
Wi g. P*) + KR.
Proof. We prove Wi(Pg, PM) < 6 + eg each round t, where subset of clients S, S S is sampled.
The Wasserstein-1 distance is:
WiPP, Q) inf 7EI((P.)) / || 201==1zzzz d7(2,2')
First, we prove the bounded local private feature distribution to the global surrogate distribution distance. For each client & E S (not necessarily in the sampled subset St), we aim to bound the Wasserstein-1 distance between the local feature distribution Ps and the global surrogate feature distribution Ps, To do we introduce the local surrogate feature distribution P intermediate
25

---

distribution and apply the triangle inequality for the Wasserstein- distance. The triangle inequality states that for any three probability distributions, we have:
W (PK., P") S W (Pa, Pk) + Wi (PK,
This incquality decomposes the distance between Pk and pa into two segments: the distance from the local true feature distribution Ps to the local surrogate Pi. and the distance from the local surrogate Pž the global surrogate p Now, we have (1) The condition W1 (Pi. Pk) I implies, by the symmetry the Wasserstein- distance (W1(P, Q) = (i(Q, P)), that:
Wi (PKk,, PÊ) (PP, PA) E1.
This bound measures the alignment quality between the local features and the surrogate features for client k, reflecting how well the surrogate data approximates the true data in the feature space. (2) The condition W, (Pi, P*) S 62 directly provides:
Wi (P%, PT) C2-
This bound measures the consistency between the local surrogate features and the global surrogate features, reflecting the uniformity of surrogate representations across clients. Substitute these bounds into the triangle incquality:
W1 Wi(Pk, P*) < Wi (P P*) S t
Thus, we have:
(P PT) E t
This bound holds for all S, as the given conditions apply to each client, and the global surrogate P% is defined over clients.
distribution = K Σボ= p==
Additionally, we denote the Pg,x as the feature distribution on the local data distribution Dr by the global feature extractor vb. Then, we build a connection between Pg,: and p By Lemma A.4 and (9), we have:
W1 9. P*) Wi 9.k Pk) (P P*) K k +
(10)
Lastly, since Pg the feature distribution ub over the global data distribution, equivalent the average of Po, construct = * E Thk where E 11(P g.k: P*). Marginals:
First: [4 x22,2 = 1 K Ek= pg, = Py
Second: z3)(2,z²) = & Σ pa = P
The cost is:
1 K K
f1I= 2' (12
Wi W1(P g. PP) Sk+ee+e
k=1
Thus:
WP P*) S e2 +
which completes the proof of Theorem 4.1.
B
More Facts about FedGPS
In this section, we outline some facts FedGPS, including the communication overheads (Sec. B. and privacy issue of FedGPS(Sec.BB meaning of Nemenyi post-hoe test (Sec. B.3). We also provide a theoretical justification dynamic path-oriented rectification (Sec. B.4).
Furthermore, we also give brief introduction about the
26

---

Table 5: Detailed description of the two consecutive upload and download rounds, along with the associated local computational requirements.
Process
FedAvg
FedGPS
1. Update global model d' 6 R, 4t-
Global aggregation round! --
Update global model # 2. 222
Update global sunogate protosypes 2 4
Explanation
Apurt from the direcct agaregation panmoters, e.g.. FedAvg-like like. The client parameters has also been widely used in many stodies.
Server
Select subser Sy to purticipate Round
Global model dt; Global model infonmation AP
Round selected S, download from server
Global model (# Comm M)
61 by Exes,,, Ag" (A Comm C+ 512)
Af
iggrogation infocmation updated selected clicnt previeus round
Explanation
plobal surogate distributiion represented by protorype for 512 dimensional vector.
The prototype class
Local operation
Uplate local model #i # (0 menss the model without
Updane local model # 8 Compute Nos-Self Grudient besed on A
epuchs training)
A: 4 28 & If this clieni is selected last round which means we should disirec its local update 25;" of last round (this is kept locally).
Local extra operation explanation
B: - 50 this client not selected last round which means A0' consiin all other client's gradiemt infocmation.
Local training
Traditina SGD the corrsponding SGD use Ey, (6) FedGPS with local data function local data
Incurporating gradient information other clients only occurs adking purameters logether
Explanation
gradient descent. This additional computation overhead almest negligible and can diseegnned. total of Resation limes paranctor vommation opentions. (Nogligihic additional computtrivn expessos)
1. Local updated parameters D5 0 0,e Compute local surrogate protonypes f Ton Eeu(s)) (e Comu
Round selected uphodd surver
local model punmeters Comm M)
M 4 C+ 512)
Glohal aggregation at round:
Σ %
#1*1 H' + my Exes, 4
We denote the whole model size as M and the total clases of the datnset is C.e..C 10 for CIFAR-1O.
Table 6: The performance comparison between FedGPS-CE and FedGPS under Heterogencous scenario 1 with CIFAR-10.
K- = 10 = 50%, R
= 500, E =1
K = 10, A. = 50% R
= 200, E =
K 100, = 10%, It = 500, E
FodGPS-CF
90.01
88.13
78.07
FodGPS
90.31
88,47
78.32
B.1 Communication Analysis of FedGPS
In this subsection, we give a detailed communication overhead analysis regarding FedGPS. In summary, there is one additional model (containing the aggregated gradient) of the same size as the global model during the download stage, while no extra communication overhead during upload from the client to the server side. Thus, FedGPS brings about 1.5 times the communication overhead than vanilla methods, e.g., FedAvg. Specifically, we have detailed the computational costs of the server and client, as well as the communication costs of download and upload between two adjacent rounds, and explained the reasons for these additional costs of FedGPS in Tab. 5. The extra C' * 512 (where C' * 512 & M, e.g., 0.05% in CIFAR-10) is the local uploaded local surrogate distribution prototypes and download global surrogate distribution prototype.
We also tried a communication-fiinn friendll version of FedGPS. which was denoted as FedGPS-CE Specifically, when uploading the model, FedGPS-CF, like FedGPS, only has a communication cost of M + C 512. When downloading from the server, it still only downloads the global model and global surrogate prototypes, meaning the communication cost remains M + C' 512. Here, we use the difference between the global model downloaded in two rounds from the server to represent the gradient aggregation information of other clients. Similarly, if a client is selected in both adjacent rounds, its own update information should be removed. Finally, FedGPS-CF achieves comparable performance to FedGPS; the results are listed in Tab. 6. Moreover, it reduces the download overhead of M, making FedGPS-CF only have an additional communication cost of C 512 compared to FedAvg, which is negligible.
27

---

B.2 Privacy Analysis of FedGPS
The two technologies of FedGPS do not transmit any additional client's information to any other client compared to traditional methods. On the contrary, FedGPS replaces raw data prototypes with surrogate prototypes instead, thereby further protecting privacy compared to previous works [31, 61]. We list the privacy clarification among surrogate distribution and gradient information in the following:
Regarding the surrogate distribution privacy issue: Because the surrogate is sampled from different Gaussian distributions, it does not contain any information related to the local data. Further, we transmit the aggregated class-wise prototypes of surrogate data, After aggregating all the high-dimensional embeddings of cach class, the surrogate data information is further protected. Many papers [31, 61] also transmit using the original data prototypes, which is weaker level of information protection than FedGPS.
Regarding the gradient information privacy issue: FedGPS doesn't I transmit the gradient information of certain client to any other client. Every client only upload its own infor- mation to server and download the aggregated information from the server. This process is the same most of other federated methods in uploading and downloading the aggregated information (Detailed information can be referred at the Tab. 5).
B.3 Brief introduction of Nemenyi post-hoc test method
The Nemenyi post-hoc test in Fig. is a non -onaammmettiic statistical method used for pairwise compar- isons of multiple groups [62] (e.g., algorithms or models) after a significant result from Friedman test (a non- panpaaametric analog to repeated-measures measures ANOVA). It ranks the performance of each method across multiple independent runs or datasets and computes "critical distance" (CD) thresh- old. If the average rank difference between two methods exceeds the CD, their performances are considered statistically significantly different at given significance level (typically a=0.55). The test is conservative and accounts for multiple comparisons to control the family-wise error rate, making it robust for scenarios like ours, where we evaluate algorithm robustness across diverse heterogeneity partitions (e.g., different random seeds for Dirichlet distributions). In Fig. 1, the Nemenyi post-hoc test assesses the robustness of bascline methods across different beterogeneity scenarios. The results show overlapping CD intervals for most baselines, indicating statistically significant performance differences among them. This finding highlights the need for our proposed approach, as existing methods exhibit limited adaptability to varied data distributions. Furthermore, the Nemenyi test has been widely adopted in holistic evaluations [63, 64, 65) and federated learning scenarios [66, 67].
B.4 Theoretical Justification of Dynamic Path-oriented Rectification
The insight behind incorporating information from other clients in FedGPS stems from works like SCAFFOLD [12] and other related research [18], which also use other client information as a control variate to adjust update direction. Beyond this intuition, we provide theoretical justification using a Taylor expansion to demonstrate that integrating other clients' information can indeed further decrease the deviation between local and global update directions.
We denote local loss function as f+(() Rd + IR and global loss function F(-). First of all: The original local update use the vanilla gradient descent on the local model O₂ is denoted as
6g,
Jold (O,)). For FedGPS, we incorporate non-self gradient dø, to local model, we denote Th II as g
0% = e + Azg'-
Then get new update direction computed based on the new model parameters 0%:-
Jnew V (0%))
For continuously twice-differentiaall function fu (0). the gradient V expands around point 0 along direction g
(O + (agnk) R (a((O2)) + V (ax) + Ra.
where R3 represents higher-order terms that can be neglected and V2 fa is the Hessian at Og. Thus we can get:
V (()) ~ (0x) + X fi
28

---

Here, we assume that the loss function is convex. So the Hessian H = V is positive semi- semi-definit.. Ideally, we assume non-self self gradients contain all the gradients from other clients; we can denote do, -V)(0) (. Substitute into the expansion:
(fa( = ( A(FF Vf((0)) - -I (I- ( +
+gg
where / is the identity matrix. As result: The original bias between local and global gradient: do = Ilgaia (0) Refined update direction bias between new model parameters 0% and global gradient: d' le V (OO)) ~ [|(I (V VF(O))] S || X do- If X is tuned make || X H 1/1, then d' do. which reduces the shift.
In practice, direct access all client gradient information is often limited due to privacy and communication overhead. Nevertheless, through careful hyperparameter tuning, FedGPS consistently achieves state-of-the-ard (SOTA) performance across various heterogeneous scenarios.
C More Experimental Details
this section, we present comprehensive overview of our experimental implementation and process. First, we visualize various data distributions across diverse heterogeneous scenarios (Sec. C.I). Next, we provide the hyperparameters
both the baselines and FedGPS (Sec. C.2). Additionally, this section includes the process and pseudocode for FedGPS (Sec. C.3).
C.1 More Data Distribution
We present detailed visualization of different data distribution across various datasets and client numbers with the same heterogeneity degree a 0.1. A heatmap visualizes the distribution, with darker colors indicating higher quantities for the corresponding label.
Figure 5: The visualization CIFAR-10 dataset distribution across K clients under five different heterogeneous scenarios.
Figure 6: The visualization of the SVHN dataset distribution across K 10 clients under five different heterogeneous scenarios.
The Figs 5, 6, 7, 9 and 10 demonstrate that, despite using the same dataset and degree of
heterogeneity, the data distributions across different scenarios vary significantly. This variation can lead to differing performances of the same algorithm. Therefore, the algorithm's 's robustness across various heterogeneous scenarios is crucial.
29

---

Estergation voceate
Figure 7: The visualization of the CIFAR-100 dataset distribution across 10 clients under five different heterogeneous scenarios.
Taavogoiillan scmaaio
Figure 8: The visualization of the CIFAR-10 dataset distribution across K 100 clients under five different heterogeneous scenarios.
06) Hddeggeiia
(4) Merangg RiT acoradoo
[6] Basrrgiiiess acciurto
Figure 9; The visualization of the SVHN dataset distribution across K 10 clients under five different heterogeneous scenarios.
[A] Thairopllkke coenato
06) Hocnggheee 2
Tirning lides scaurio
16 Haiiogiill searre
(e) Hanire soearad
Figure 10: The visualization of the CIFAR- 100 dataset distribution across K 100 clients under five different heterogencous scenarios.
30

---

Table 7: All the hyperparameters of the compared baselines, including learning rate, momentum, weight decay, Nesterov, and the remaining hyperparameters the method itself.
Mothod
Learning rate
Momcotum
Weight docay
Nesterov
Other hyperparameters
FedAvg
0.01
0.9
0.00001
False
None
FedAvgM
0.01
0.9
0.00001
Truc
Monentum coefficient: 0.9
FedProx
0.01
0.00001
False
Proximal coefficient:
SCAFFOLD
0.00001
False
Global step
CCVR
0.001
0.9
0,00001
False
None
VIL
0.01
0.9
0.00001
False
VHI alpha:1.0
FedASAM
0.01
0.9
0.00001
False
Rho: 0, eta: O
FedExp
False
Eip:: le-3, eta_g: 1.0, lr. Se_weeeh__eeee decay: 0.998
FedDecor
0.01
0.00001
False
Foddecorr coefficient:
FedDisco
0.01
0.9
0,00001
False
Metri: KL divergence' feddisco 0.5, feddisco b: 0.1
FedInit
0.1
None
0.001
False
Beta: 0.1
FedLESAM
0.1
None
0.001
False
Rhor0.5, betar0.1, max 10rm:10.0. global step size:1.0
NUCFL
0.001
Calibration DCA, uniform penalty:
FedGPS (Ours)
0.01
0.00001
False
A 0.1, Ag 0.2, X, 0.5. 1.0
Algorithm Pscudo-code FedGPS
Server input: communication round T, server initialize the model 0
Client k's input: local epochs E, k-th local dataset T
Initialization: all clients initialize the model 0% and surrogate data D
Server Executes:
for each round /. = 1,2, ..... do
server random samples a subset of clients Se S K. server communicates 0f to selected clients for each client & E S, in parallel do
A++1 Local Training (k, 0 end for 0--1 = 0² 1 7l |S ExEs, 4+1
1
end for
Local
Local_Trainee Training k: 0¹):
Update local model by global model 0% + 0r
Compute 5% using Definition 4.2
for each iterations = 1.2, do
Compute the g 1+1,e+1 new gradient using Eq. and Eq.
Update local model at e iteration: g 1,++1 end for
k 1,e
1,441
Compute the update information this round for client k: A = 00+1 O
Storage A (+1 for Non-self gradient computation k Return ++1 to server
C.2
Detailed Hyperparameters
We list all the hyperparameters of the baselines and our framework FedGPS in the Tab. 7.
C,3 Process and Pseudocode of Algorithm
In the Algorithm section, we elaborate on the pseudocode workflow of FedGPS Algorithm Consistent with other federated learning (FL) frameworks, predefine the number of communication
31

---

Table 8: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on SVHN, heterogeneity degree a = 0.1, local epochs E = 1 and total client number K 100.
Datsor SVHN Hoteoggacity Levetre
0.3 Cltcst Number R 100, Clion Sampling Rate 10% Thxal Communicaton Roush
500 Local Epects X
Scasado
Werrnngerioee 3crtoriii 2
Heiteggaass
Heinogroons somartio
Cramrilaad Threning AnzVir
ADC ROUND j SpootUy
ACC ROUND j Speollip
ACCT BOUND & Speodlipt
ACCT ROUND a SpeedUpt
ACC1 ROUND,
Speedlipi
Mettods
Tareet Acc
Tagett Acce819
Tageei AS
Tagast Acc-O155
Tlager, Acce01
Meun Acet Std
Podlog
91.15
478
1.3k
91.05
450
51.94
238
1,0x
91.07
617
1,0X
91.17
200
1.0x
91,40 0.39
PedhogM
02.06
308
1.3x
02.53
318
1.4x
101.30
310
1.2x
50.30
097
0.30
91 13
107
1180%
92 31
+ 0.67
RedPics
10.02
None
91.73
1/0x
696
0.8%
Note
Nose
SCAITOLLD
00:15
300
01 86
1.4%
00.07
015
0.5
99
131
100
95.10
92.
CCVR
91.24
107
4.7
02.02
102
4.0%
01.71
104
4.5%
91.01
93
4.6%
90.00
185
285
91.66 2 0.45
VHL
00.47
50.70
19x
235
1.8x
205
94.00
93.50 =
PMAAAM
10.54
Novs
01.13
00.50
390
140
131
92:11
0.5%
91.
0.71
PedEsp
10:93
Nooc
Nove
10.73
No0e
No00
10.11
696
0.8x
91.41
None
Note
91.40
201
1.0%
93.02 6 0.54
PodFoarr
01
Nevs
Nove
01:15
479
1.0
01.83
None
Nen
None
Nore
90.07
Nen
Nare
91:00 0.11
FolDico
Fedlnx
14.13
259
1.5x
50.45
233
18K
54.25
238
1,6x
14.28
203
1.5x
94.00)
820
1.2%
94.04 0:033
Fedt BSAM
64:07
911
7.36
01.83
160
3.0x
04.03
901
1.5x
94.07
191
9.30
91.78
187
8.1x
91.71 0.000
NDCFL
[4]0]
None
01.05
1
01.25
Mone
None
None
Mode
91.41
0.:13:=0.31
OelG
955.03
915
3,3x
85.37
3.7x
05.14
230
1.0x
R5.01
223
05.00
108
2.0x
85.08 0.007
Table 9: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on CIFAR-100, heterogeneity degree a 0.1, local epochs E and total client number K
100.
Dunect: CIPAR--00 Hesoogemnnty Laszbo
Chen Numban
100, Clicri Sumplnn Raic 10% Communicaiion Bourd T 500 Local Epucht
Dill Sczsario
Hatengeszous
Hetorgeesotas
Hetorggmmeoss sccrario
Halooogenors socsario 5
Cromallad Theening Ares
ADC ROUND j SpoedUyr
ACC ROUND Speetlyr
ACC ROUND 1 Speedl
ACCT ROUND i Speediyr
ACCY ROUND, Speedlipt
Mathos:
Targer
Tergrt Accust/
Tirgrt Accut1S
Target AccudtS
Tirger
Pelligg
6479
481
1.0x
13:40
500
1.0x
11:55
1.0xx
12.00
089
1.0
35:17
1.0%
41.00 + 4.56
PedAogM
47.22
457
1.3x
09.53
353
1.3x
01:11
373
1.3x
50.01
340
1.4x
06
507
2.1x
1.42
Bolthes
SCAFFOLD
63.04
Noxz
None
43.34
416
LiXx
41.72
685
1.0x
41.30
Noxc
Nose
40.14
502
144
41.02
1,30
CCNR
FMdANAM
Nove
09
1.3x
40.00
1.1x
632
(1 2.51
Ballep
FoiDesor
47.26
427
64.06
LEX
40.01
424
L.x
201
1.3x
45.03
46.16 :
FedDisco
Peelive
59.09
351
1.4%
57.05
336
1.5%
50.59
330
1.4x
36.00
331
1.5x
55:00
257
1.k
56:15 4 0.95
RMLESAM
57.78
277
1.7M
57.17
200
1.9k
55.07
354
1.8x
36.65
577
1.8k
56:14
811
23x
56.56 8 1.03
NUCFT.
FedGPS:Oun)
68.77
264
1.8x
67.84
362
1.8%
55.89
240
1.9K
67.62
200
195
57.
207
244
67. 6: 1.00
rounds and initialize both global and local model parameters. In each round, a subset of [Sr| clients is sampled for local training and model weight communication. Distinctively, FedGPS first computes the non-self gradient locally. This non-self gradient is then used derive new weights, corresponding to the dynamic path- nath-oriented rectification process. Subsequently, using these new weights, new gradient direction is obtained via Eq. 6, which aligns with the static goal-oriented learning objective. The original parameters are then updated based on this new gradient direction. We highlight the key differences from the vanilla FedAvg algorithm to underscore the unique contributions of FedGPS All related hyperparametrs can be referred to Tab. 7.
D Further Experimental Results
In this section, we perform additional ablation studies to demonstrate the effectiveness of FedGPS. We first compare with more baselines (Sec. D.1). Then we explore various settings, including different numbers of clients (Sec. D.2), varying local training epochs (Sec. D.3), various client sampling rate (Sec. D.4), different degrees of heterogeneity (Sec. D.5), another heterogeneity partition method (Sec. D.6), and multiple training seeds to ensure robustness against variations in training initialization and procedures arising from random client sampling (See. D.8). Lastly, we give some visualization of the whole training process to verify the performance and convergence (Sec. D.7).
32

---

Table 10: The results comparison between FedGPSand more baselines under Heterogeneous scenario 1 with different datasets.
K = A0 50% R 500, E
K 10, = 50%, R 2000, E
= 100, 10%, Rt
5000 E
CIFAR-10
FedAdam [68]
85.34
85.08
67,43
A-SGD [69]
86.96
86.66
69.49
FedMR [41]
TedGPS
90.31
88.47
78.32
FodAdam [68]
89.80
93.27
A-SGD [69]
89.78
90.21
94.08
FedMR [41]
91.32
86.64
FedGPS
94.20
CIFAR-100
[68]
A-SGD [69]
70.07
67.78
57.48
FedMR [41]
69.45
67.45
FedGPS
71.14
68.90
58.77
D.1 More Baselines
Besides, we also compare with other strategies to mitigate the heterogeneity problem in FL. Firstly, we compare with adaptive methods. We select two representative adaptive methods [70], e.g., A- SGD [69] and FedAdam [68]. Furthermore, we also include the FedMR [41], which is new method to modify the aggregation strategy [71]. The results are shown in Tab. 10, FedGPS still outperforms these methods.
D.2 Ablation Study on Client Number K
In this section, the results are listed on Tabs. 8 and 9. We extend our evaluation of FedGPS beyond the CIFAR-IO dataset to include the SVHN and CIFAR-100 datasets, focusing on scenarios with large number of clients (simulating cross-device settings). Specifically, randomly sampled each round for local training with E = 1 local epoch, followed by aggregation. Experimental results reveal that, compared to the 10-client scenario, the 100-client setup with lower sampling rate leads to reduced model performance within the same number communication rounds. The SVHN dataset, owing its relative simplicity, exhibits minimal performance degradation. In contrast, the CIFAR-106 100 dataset experiences more pronounced impact. Furthermore, several baseline methods struggle to converge when training larger models, such as ResNet-50, on CIFAR- 100 with a low sampling rate, often requiring extensive hyperparameter tuning to address these challenges. Further experiments additional client experiments, e.g., 500 clients across the entire FL system, as shown in Table 11, The results still show the superior performance of FedGPS. Beyond the ResNet-based model, FedGPS also consistently shows improvement on ViT-based models, as Tab. 12 shows.
set 100 clients, with 10%
needed investigate these issues thoroughly. We further conduct
Table 11: Experimental results of larger number under Heterogeneous scenario with CIFAR-10 dataset.
Table 12: Experimental results of ViT-based model for 100 clients under Heterogeneous see- nario with CIFAR- 10 dataset.
K = 500, A, 10%,R=500, E=1
K 100, A,
10%,R= E=1
FedAvg
73.19
FedExp
74.48
FedAvg
30.45
FedExp
28.35
FedAvgM
75.43
FedDecorr
74.24
FedAvgM
33.56
FedDecorr
39.63
FedProx
76.72
FedDisco
FedProx
48.46
FedDisco
SCAFFOLD
64.18
FedInit
73.05
SCAFFOLD
35.38
FedInit
32,34
CCVR
64.43
FedLESAM
80.67
CCVR
Fedi RedLE
45.83
VHL
80.58
NUCFL
73.98
VHL
47.36
NUCFL
38.89
FodASAM
75.06
FedGPS(Ours)
82.18
FodASAM
35.43
FedGPS(Ours)
56.71
33

---

Table 13: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on CIFAR-10, 10, heterogeneity degree a 0.1, local epochs E = 5 and total client number K = 10.
Ditruct: CIFAR Hettongacky Lavelos
Clicni NumberKK
Clizn Sampling loarr 56% Communicaan Rranc:T
Local Epochs:
Dill Sczsario
Hetengennose
Hetarogmasss
Holeoogmmors socsario
Centriliad Traini Accelóia
ADC ROUND j SpeedUyr
ACC ROUND Speedlyr
ACC) ROUND 1 Speedlyr
SCCT ROUND 1 Speediyr
ACCT ROUND Speedtipt
Method
Teenci Acce
Tigact Acce
Tapasi Acce725
Acced5i
Maso
Std
Petfeg
85.59
192
1.0x
55.35
136
1.0k
60.51
111
1.xx
71
91
100x
80:51
151
1.0%
75 TO 8:17
Fellagh
65.40
118
1.1x
67
1.4x
20.55
75
2.0
FodPecs
188
54.31
Nore
No0e
57
74.00
50.00
SCAFFOLD
51.75
Nov
Nove
50 30
Nore
No00
14
Noce
Nexo
70
90
10k%
71.14
47
75 50 6 4.75
CCVR
53.93
Nove
No0r
52.87
Noce
Novc
52
Noce
No0e
78.
21
4.3%
72.36
15
103x
53 01
= 3.85
PsdASAM
65.29
118
1.5
80
12.12
135
1
01.55
None
Noxz
11.91
67
1.4g
01
75
2.0
70.26 8 7.24
PedEsp
55.05
138
1.5xc
55.94
135
10xx
62:47
Nore
No0c
74
67
144%
73.00
67
2.8x
00.34 = 5.96
FeDDeor
None
$1.90
No00
Nore
Nono
75
5.0%
FoDDico
85.00
$5.29
1/0x
146
1/0x
None
66.70
75.06 9:3.0
Pedinee
79:33
Nows
Nome
71:43
Non
Novs:
75.75
Nome
Newn
51.05
None
Nore
52.67
Nete
Nare
70.06 4 8.16
EaALESAM
90.36
164
60.94
Novz
Nonc
$1.33
None
Nomz
55.83
None
Nore
64.00
Nonc
Norc
75.63 2
NDCFL
78.45
76.51
Nose
Noce
36.90
Noxe
64.07
73.50
VetGP%:Ouno
88
65
2.8
87 56
68
2.0%
65.70
1.40
1.0%
61 68
17
19x
77 70
41
3.54
81.55 02 4.32
Table 14: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on SVHN, heterogeneity degree a 0.1, local epochs E 5 and total client number K 10.
learrnmazit
Numbork
Sampling
Communicaiion
Local Bpoche
Dall Scanario
Heszmngennns a
Nateenganzon 2
Heserngmmoee sezzario
Helergganoes aconario
Heteoogronn socnario 5
Contaliand Trraning Aucrit
ACC ROUND, & Upo
ACC ROUND Speollyr
ACCT ROUND Speedlyr
ACCT ROUND, à Speedlyr
ACC1 ROUND,. Speedlipr
Metbods
Tageet Acce915
Tagget Acc-926
Tagest Aco-025
Menn Sid
Ped//g
81:21
167
1,3x
61.99
103
L.0x
12.51
100
1,0x
32.05
123
1.0x
07
CD
1.0x
91.77 = 1.45
NeHAugM
Fallbos
63.81
Nenz
None
91.00
1.2x
50.52
1.2x
18
Noxc
Nore
95 22
52
1.2x
00
L.3
SCAFFOLD
63:42
Nonz
Nom
50 33
Nom::
Nonc
01.53
Nonc
Nexz
92 15
103
1:3
91.61
Nevz
Navc
59. 00
3.46
85.06
None
24
Nove
Nore
Nose
Noxe
Nose
Nose
11:41
Nosc
VHL
58 50
1.6x
94
93.
PrA4AM
W1.66
Nevz
Nom
22 01
71
1.4%
10.59
71
1.4g
92 19
131
0.0
95. 30
61
1.0%
91.76
9.71
FolEap
63.31
None
52.4 46
100x
1.4x
92.08
165
0.0k
91.67
RadBesso
Non:
130
FodDico
63.43
Noxe
None
91.72
100
1.8x
12.53
59
18xx
92.27
97
1.4%
12.50
64
1.0x
91.50 8: 1.80
Fedinx
67:26
None
Noxe
77.42
Nooe
None
50.57
Nore
None
55.07
None
Noce
55.06
Nore
Noxe
08 10
FSAM
NDCFL
80.65
Nexz
None
50.30
None
None
50.75
Nose
Nonz
91.21
Noxe
Noze
91.68
Nose
Nuzc
90.00 2: 1.88
OulG
95.651
50
2.14
94.20
57
1.0%
65.06 08.08
49
2.0
94.30
68
2.0
94.76
50
1.5
94.30
1 0.36
D.3 Ablation Study on Local Epoch E
The number of local training epochs significantly impacts the performance of federated learning (FL). Excessive local fitting can exacerbate performance degradation. To investigate the effect of local epochs on various existing methods and FedGPS, we adopt a consistent experimental setup with K 10 clients, a heterogeneity degree a = 0.1, and 50% of clients randomly sampled per round, varying only the number of local training epochs E 5. Experiments are conducted across the CIFAR- 10, CIFAR- 100, and SVHN datasets, with results reported in Tabs. 13, 15 and 14. The findings indicate that performance generally declines as local epochs increase for most methods, including FedGPS, VHL, and FeMLESAM. For the methods with increasing performance, especially for some local learning objective modification methods, the additional penalty term increases the learning difficulty, and a small local epochs cannot fully train the local model, so a larger local epochs is more suitable for such methods. However, the larger local epochs make the performance of these methods after over- wer-training still needs to be further verified.
Observation 4: SAM-basee method needs to be adapted to different local epochs. SAM-based methods require distinct gradient perturbation strategies depending on the number of local epochs. Notably, FedLESAM, SAM-based method, exhibits significant performance degradation with
34

---

Table 15: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on CIFAR-100, heterogeneity degree a 0.1, local epochs E and total client number K = 10.
CIFAII-1O Besnogmnelly Lavclo 0.1 Namber
10, Sumpling 50% lotal Communattion Bosndl/
Epuche
Sczsario
Helongennots
socsario
Centridiaad Trainny Accetii
ADC BOUND & SpeedUyr
ACC ROUND Speedlyr
ACC) ROUND 1 Speedlyt
SCCT ROUND i Speediyy
ACCY ROUND Speedtipi
Methodi
Acce57%
Accen7ti
Tiract Acced 7%
Accediiii
Acce64ti
Meso
Pelfag
07.95
138
1.3x
07 72
176
1.0x
07.50
111
1.0x
03
181
100
$4.27
170
100x
67.38 1.70
Fallagh
69.85
1.0
180
00:40
FodPecs
Nore
No0e
17.74
Note
60.30
SCAFFOLD
61.31
Nove
Nove
12.15
None
Ne00
12
Noce
Noco
04.11
None
Nore
31.01
Nore
Nore
63.02 1.45
CCVR
PsAASAM
69.23
110
1.3%
68.30
1
68.00
186
1.0
69
176
52 55
Nonc
Naxe
67. 52 2.51
PedEsp
68.20
123
1.3x
67.00
170
100x
08.07
182
0,3
103
0.5%
10.00
Noec
Nose
65.11 4.35
FedDico
180
67.01
Pedine
61:20
Noms
Nome
61.34
Non
Nom
00.95
Nome
Neo
55.15
Nons
None
57.57
Nenn
Nare
60.51 4 9.00
EaILESAM
01:41
00:25
Nonc
60.14
None
Nomz
51.80
None
None
58.21
00:42
NDCFL
20.05
Nose
Nooe
None
62.00
DetGP%:Duno)
(B8 90
130
1.3%
66 85
147
1.2%
68 56
180
0.8
56
Nome
Nore
65. 71
114
1.5x
65: 28 0.60
Table 16: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on CIFAR-10, heterogeneity degree - 0.05, local epochs E = and total client number K 10.
Datuset: CIFAR- 10 Heterogeneity Levelto
0.05 Client Number: 10, Client Sampling Rate: 50%
Total Communication Round:T 500 Local Epochs:::
Diff Scenario
Heterogeneoos scenario
Heterogeneous scenario 2
Centralized Training Acc=95%
ACC1 ROUND SpeedUpf
ACC ROUND J. SpeedUpt
Methods
Target Acc=
Target 49c
Mean Accit Std
FedAvg
75.28
298
1.0%
45.59
233
1.0%
60.44 + 20.99
FedAvgM
75.81
105
2,8x
19.26
275
0.8
62.53 + 18.77
FedProx
79.77
176
1.7x
52.17
237
1.0x
65.97 + 19.52
SCAFFOLD
46..67
None
None
51.14
180
133x
48.91 + 3.16
CCVR
78.20
94
3.2x
60.53
40
5.8
69.37 1: 12.49
VHL
85.22
143
2.1x
69.09
84
2.8x
77.16 + 11.41
FedASAM
75.84
198
1.5x
50.62
275
0.8x
63.23 + 17.83
FedExp
75.70
198
1.5x
46.86
None
None
61.28 1: 20.39
FedDecom
81.03
105
2,8x
51.09
2.4 2.4x
66.36 4: 20.75
FedDisco
82.37
1.7
51.01
110
2.1x
66. 08.69940 + 22.17
Fedinit
77.55
474
0.6x
47.99
None
None
62.77 + 20.90
AM
79.92
198
1.5x
51.90
2.3x
65.91 +: 19.81
NUCFL
72.96
None
None
16.92
None
None
59.94 18.41
Fud))S(Ours)
86.97
1.5x
70.48
2.5x
78.72 + 11.66
increasing local epochs across all three datasets (SVHN, CIFAR-10, and CIFAR-100), In contrast,
FedASAM, another SAM-based method, shows minimal degradation and, in some cases, slight improvement. This suggests that SAM-based methods necessitate tailored gradient perturbation mechanisms when local epochs are extended.
D.4 Ablation Study on Client Sampling Rate X
To investigate the impact of client participation on the performance of FL. systems, we conduct an ablation study the client sampling rate. In FL frameworks like FedAvg [7], the sampling rate
35

---

Table 17: Top-1 accuracy of baselines and our method FedGPS with 5 different heterogeneous scenarios on CIFAR-10, heterogeneity degree 0.05, local epochs E and total client number K 100.
Dataset: CIFAR-10 Heterogeneity Levela 0.05 Client Number:K 100. Client Sampling Rate: 10%
Total Communication Round.T Local Epochs:
Dift' Scenario
Heterogeneous scenario
Heterogemeous scenario 2
Centralized Training Acc=95%
ACC1 ROUND 4 SpeedUpt
ACC1 ROUND J. SpeedUpt
Methods
Target Acc=32%
Target Acc=34%
Mean Acc Std
FedAvg
32.97
164
1.0x
34.36
473
100x
33.67 0.98
FedAvgM
41.86
256
1.8x
33.15
None
None
37.50 6.16
FedProx
36,63
211
1.9x
31.99
486
1.0%
35.81 1.16
SCAFFOLD
48.80
27
17.2x
50.16
31
15.3x
49.48 =: 0.96
CCVR
55.28
28
16.6x
59.36
24
19.7% x
57.32 = 2.88
VHI.
59.79
115
4.0%
50.85
163
2.9x
59.82 + 0.04
FedASAM
33.35
177
1.0x
30.14
None
None
31.75 = 2.27
FedExp
None
None
32.79
FodDecom
47.85
212
2.2%
FedDisco
FedInit
52.98
57
8.1x
58.51
65
7.3x
55.74 4: 3.91
ESAM
60.90
56
8.4x
64.63
69
6.9x
62.77 + 2.64
NUCFL
10.70
264
1.8x
37.91
430
1.1x
39.30
= 1.97
Fedrs
65.86
70
6.6x
70.14
72
6.6x
68.00 + 3.03
Table 18: The ablation study of client sampling rate under 0.1, K = 100, R = 500, E = with CIFAR-10 dataset.
Sampling rate A 5%
10%
20%
50%
Sampling rate X 5%
10%
20%
50%
FedAvg 34.59
48.22
67.38
72.45
FedExp
34.38
38.26 58.35
64.52
FedAvgM
35.32
58.80
68.34
73.89
FedDecorr
63.69
77.31 80.03
FedProx
32.89 52.84
65.25
74.24
FedDisco
SCAFFOLD
36.73 60.17 67.59 69.34
FedInit
30.56 71.01 76.89 77.59
CCVR
57.93 64.06
74.25
78.78
FedLESAM
68.86
72.64 76.35 76.90
VHL
57.14 72.70
76.58
80.78
NUCFL
38.85 53.72
68.31 71.91
FedASAM
34.78 46.35 61.84 66.48
FedGPS(Ours)
70.89 78.32 79.72 81.97
determines the fraction of clients randomly selected per training round, balancing computational load, communication overhead, and model convergence. Lower sampling rates reduce bandwidth usage and enable scalability resource-constrained environments, but may slow convergence due to noisier updates from fewer participants. Conversely, higher rates accelerate learning at the cost of increased synchronization demands. By varying the sampling rate (e.g., from 10% to 50%) while keeping other hyperparameters fixed, this experiment isolates its effects on metrics such as test accuracy. The results are shown in Tab. 18. Under different client sampling rates, FedGPS still performs better than other basclines.
36

---

D.5 Ablation Study on Heterogeneity Degree
To assess the impact of varying degrees of heterogeneity, conduct experiments under constrained computational resources. Specifically, we evaluated two distinct heterogeneity scenarios on the CIFAR-I0 dataset, with setups of 10 and 100 clients, respectively. The results are presented in Tabs. 16 and 17. The results demonstrate that as heterogeneity increases, overall performance declines, and the performance gap across different heterogeneous scenarios widens. Specifically, as shown in Table 16, FedAvg's accuracy drops significantly from 75.28 to 45.59, a decrease of 29.69, Notably, FedGPS exhibits greater performance improvements in more challenging scenarios. For instance, in the a = 0.1 scenario, FedGPS outperforms the best baseline method by average of 1.04 in accuracy, while the more heterogeneous a = 0.05 scenario, this improvement rises 1.56. The advantage of FedGPS becomes even more pronounced with larger number of clients and lower sampling rates, achieving an improvement of 2.06 in the = 0.1 scenario and 5.23 the a 0.05 scenario shown in Tab. 17. These findings further validate the effectiveness and robustness of FedGPS under diverse and challenging FL conditions.
D.6 Different Heterogeneity Partition Strategy
Table 19:
Experimental results under = heterogeneity partition method with CIFAR-10 dataset. Here we mainly select C = 2 and C 3 these two scenarios.
C C' c==3
C' c=3 C C=
FedAvg
51.75
69.97
FedExp
47.53
67.29
FedAvgM
50.61
73.21
FedDecorr
69.36
79.60
FedProx
49.64
68.96
FedDisco
SCAFFOLD
55.08
74.34
FedInit
54.80
71.12
CCVR
55.83
73.69
FedLESAM
66.62
79.42
VHL.
73.53
84.16
NUCFL
50.03
72.89
FedASAM
55.33
69.37
FedGPS(OOrs)
78.17
85.71
Besides the Dirichlet distribution based partitioning, another common approach is label distribution skew with limited classes per client, often denoted as C = [72]. In this method, each client is restricted to samples from only N distinct classes out of the total available classes in the dataset, creating extreme heterogeneity. This partitioning simulates scenarios where clients have specialized or siloed data, such as different devices capturing only certain categories (e.g., one client with images of cats and dogs only, while another has birds and fish). It emphasizes qualitative skew (absence of entire classes) rather than quantitative skew (imbalanced sample counts per class). To verify that FedGPS is still robust under other heterogeneous partition methods, the experimental results in C 2 and 3 scenarios are shown Tab 19, indicating that FedGPS still robust to different heterogeneous partition methods.
D.7 More Visualization of Results
In this section, we visualize additional experimental results, which illustrate the dynamic training process while highlighting performance and convergence speed. Due to the large number of baselines, we visualize only the top-5 methods in this setting alongside FedGPS for comparison. The results are shown in Figs. 11, 12, 13, 14 and 15. The results reveal that in the early training rounds, FedGPS does not exhibit a significant speed advantage over other methods and, in some cases, converges more slowly. However, training progresses, FedGPS demonstrates sustained performance improvements later rounds, while other methods plateau, with their performance stabilizing. This observation motivates future research into developing more efficient variants of FedGPS.
37

---

70
Accuracy 1%
35
FedExp FedASAM
VIL
Ped.wg
FedDisco
Ours
200
300
500
Round
Figure 11: The training process visualization of top-5 baselines and our method FedGPS on CIFAR- 100, heterogeneity degree 0.1, local epochs E 1 and total client number K 10 under heterogeneous scenario
90
Accuracy [%]
60
30
FodProx
FedDecorr
100
200
300
400
500
Round
Figure 12: The training process visualization of top-5 baselines and our method FedGPS on SVHN, heterogeneity degree 0.1, local epochs E' and total client number K 10 under heterogeneous scenario 1.
D.8 Different Random Training Seeds
The choice of random training seeds impacts model initialization and the random client sampling process. To further validate the effectiveness of FedGPSin this context, we conduct experiments across the same heterogeneous scenarios using three distinct random seeds to control for this ran- domness. The results, presented Tabs. 20 and 21, reveal that such randomness noticeably affects algorithm performance, with variations in some scenarios reaching up 4:2 or more. Despite this, FedGPSconssisenttly mitigates the impact of randomness on performance, as evidenced by lower standard deviations. These findings underscore the robustness of FedGPS, not only in addressing
38

---

90
Accuracy 1%]
60
30
FedAvgM
FedASAM
VII
Fedinit
SCAFFOLD
Ours
200
300
Round
Figure 13: The training process visualization of top-5 baselines and our method FedGPS on SVHN, heterogeneity degree 0.1, local epochs E' 1 and total client number K 100 under heterogeneous scenario
90
Accuracy [%]
60
30
FedAvg
FedExp
VHI.
FedProx
150
25
50
75
100 Round
125
150
175
200
The training process visualization of top-5 baselines and our method FedGPS on SVHN, heterogencity degree a = 0.1, local epochs E' and total client number K under heterogencous scenario I.
Figure 14:
heterogeneous data distributions but also in handling variability from model initialization and random client selection process.
39

---

Table 20: Top-1 accuracy of baselines and our method FedGPS with 5 different neterogeneous scenarios on CIFAR- 10, beterogeneity degree a 0.1, local epochs
0.42
88 15 = 0.45
= 0.70
85 21 0.30
1.20
3.35
(Our)
90.31
80.52 07.67
00 20.20
88. 45
88 8U
88 00.00
77.78
87.44
85 DB
85.00 05.08
85.50 59
82.04
80.72 71
79.05
86
1.17 82
90
80
86.8888880
Datoset: CIFAR- 10 Heteregenelty Levelia = 0.1 Clieat Number: = 10. Client Sampling Rale: 30% Total Communicaioon Round R. = 500 Local Espechs::: = with 5 different rindom seets for teaining
0.91
1.61
0.50
NUCFL
60.76
62.34
81.98
79.45
77.46
80.65
=1
79:76
80.32
62.01
=
69.78
68.96
60.73
65.78
69.45
79.87
1.40 87.18 0.79 57.00 82.03
79.19
80.70
69.16
60.70
1:2.69 60.
76.
82.05 =1 1.00 87.70 # 0.51 13.10 1.13 80.30 #: 1.87 83 03.06:2 +: 0.96 82.00 = 1.15 84.16 = 2 2.30 10.000-4 -
20.26- = 00 0.88 81.02 11.02:21:000000 0.67
77:34 1.43
166 6.29
FiELES
56- 40
86,68 67.67
$4.58
.6.
85, .9.69
94.78
83 13-49
83, 50
78.00 75.09
76. 15.99
78-45
71.15
08. .5.88
70.76
1.36 71 15.25
0 9.56
9.10
Fiston
56, 02.02
88.008
80.40
800 72
82.25 27
50-45
81 $1.00
79. 219
69 44
03.87 57.57
68. 68.33
03.04
70.05
67.46
08.88
02.52
77.800
688.7 68.5
81 1: 0.32 80.31 =: 0.33
1: 1.49
7.03
Feulbec
85,60
83.30
55.75
81.84
$1.08
853.87
80.42
80.56
79.94
70.37
67.90
70.58
00.94
67.91
68.51
$4.01
80 1: 1:43 69.62
09 5.84 77.23
83,91 0.67 0.23
0.992 71.17 1-2.24 2..8
86 31
54.07
82 15
81. 18
81 .6.
73.14
72 29.8
75. 75.54
73.77
70. 01
85.
85
82
81
08.55
73
79
80.63 1 0.12 79.80 = 0.03
# 1.04 72.07 1 0.96 68 2.90
FeulEpp
84.00
84 53
70.25
79. 72
82 69
70 00
79 51
80 20
71 71.55
69.78 78
06.27
00
68 -2
1
3.39 67.71
76.03
0.70 85. 70..7001:000 U.700
6.64
FidA FRAASM
54,08
81:00
84 25
833 06
80 45
81.11
80.34 04
73.11 13
71:22
71.80
0G 08
09. 12
70.37
Turget Acc
80.
855
Target Acc- Acc-799% 79%
Target Acc- 8006
Target Acc=65%
1.15 020.72.44
26
Target Acc-
78
86.72 = 0.31
3.51
VHL
80.07
88 63
90.02
87.20
88 22
80-83
86.31
$7.00
34.30
83.07
82.08
81 31.05
80.06
78.76
and total client number K = 10under 3 different training random seeds.
25
87
83.41
79.96
89
85.
0.56 80 =0 0.63 83.77 0.20
13 =
0.144
0.84
4.30
CCVR
84.30
83.84
83.77
83.28
81.64
81.22 .2
83.00 20:00
82.23
81.87
70:07
76:31
70.52
74.72
73.49
73.11
78.47
78.77
79.74
SCAFFOLD
80.04 1.25
50.08.1 # 1.31
71 11 1,00
1.54
6,24
51 07
51.12
80.78
78.0 70.00
50.75
79.59
$1.56
69 23
72 72.03
08 08.43
65. 30
86.15
82
70
71.
75.
Msant: std 82 #: 79 83.61 # 1.71 83.38 = 0.87
81.4 = 1.13 82.93 = 0.58
= 0.06
1.00
FidPPox
88.15 1/3
86.01
86.45
83.12
84.36
82.67
82:37
63.52
82.91
70.02
77:87
77.50
08.81
68:48
07.52
77.36
68.60
7.45 79.73
00.92:4 = 0,29
2.73
Fidd vggg
85.74
84.88
84.75
81.78
85.17
83,80
81.33
62.61
50,36
70.15
69.38 69.59
70.02
67.51
71.45
66.21
7.11 77.004 à:
85.10
30
0.19 69.02
Meant: std 81 0.67
Mcant: std 80.11 0..56
1.3.... .6
Feelkng
$4.21
$3.07
$5,00
79.13
84. .0.0
50.63
10.188
79.51
08.02
08.50 50
81 .8
06 86
08.68 .5
72.50 56
53.3
Meunz std 08.07
Mean1 std 60
75
seed 0|
seed 2
seed 0|
Hoterogencuus sconario sood 1
seed 2
secd 0
Heterogeneous scomario 4 seed 1
2
sood 0|
Heltnngeeeeus scomarti 5 seed 1
seed 2
Tocal Apc std
Heterogenaaus secnario I seed
seed
Heterogennous secnario 2
E =
40

---

Table 21: Top-1 accuracy of baselines and our method FedGPS with 5 different neterogeneous scenarios on CIFAR- 10, beterogeneity degree a 0.1, local epochs
0.602
82 23 + 1.21 80,20 = 1.61 88 = 0.30
= 0.51
=: 1.61 85 03 = 0.31
2.21
14
(Our)
800 BU
85 89
87.00 77.06
85 63
$5.79
85.41
85.11 10
85,30 $5.00
77.70
73 29
75.14
87
86
84
73.777333 33.98
84.70
89
=:::::0 1.02 85.88888888
CIFAR- 10 Heteregemeliy Levelia = 0.1 Clieat Number: = 10. Client Samplin Rale: 30% Total Communication Round R. = 200 Local Especs:::: = with 5 different rindom seets for teaining
0.40
NUCFL
62.80
82.08
81.87
73.48
80.67
81.63
70.31
75.45
77.49
06.80
68:36
70.01
64.57
68.89
65,78
82.25
76.08
1.776 68
68.39
2.14 65. 37 1.0.02 63.74
78.66 61.61 i. 74.62
82.1 17 0.80
FiELES
16-36
50. 88.88
85.34
2.37 16.0
80.04
83.38 03.33
82.88 .1
$1.33
82.8 12.97
82.22 22.22
05 00-03
67. 82
00--00 NK-09
04.00
05. .2.82
05.31
47.45
Fiston
79.23
83. 01
83.05
74.40
74.25
73.64
80.08 1.37 81.56 + 0.85 84.75 = 2.19 74.11 = 0.41
75.70
71.58
75,22
0.59
0
61.05
62.87
61.93
=
2 0101
02.92
65.5 65.75
61.80
=: 7.94
51.96
97 1: 0.86 83.11 # 1.40 75.19
01.95
1.73 63.39
71.32
0:1888
1 0.80
Feulbec
85,60
85.64
55.87
01. +: 85.67
85.59
86.30
82.26
84.66
11.99
82.70
70.14
71.16
71.72
00.79
70.17
69.08
71.01
0.71 68.68
78.64
0.62
80 09.06: 1. 6.07
84.5 94.53
87 07
83 55
83 36
$1.86
83 .0
74.76
75.73
74.58
74.75
71.57
70.40
85
84
86
02.02
71 25
0.77 83.90
0.66
11211 1626
5.35
79.02 9.99
FeulExp
85.05
80:31 31
85.82
85.64
85 29
83.12 12
82 49
81.52
83.05
74.15
62
71.65
71.74
67.16
08.08
0033 88 0:29 96.04 U.222 80, 73.73
82.35
72.81
59
1.94 69.00 1.00 60
79.00
87.04 # 0.57 85.86-22 0.90
8182 # 0.28 82.30 1: 0.81
1: 2 06 75.35 0.39
6,86
FidA FadASAM
85.70
86 14
84.8 54.85
81.38
82. 19
83.022
74.91
73.63
75.51
08 01
00 08
7.01
Target AccoA5%%
86
Target Acc-85%
86.
80
Target Acc- 848
Target Acc-- 72N8
Target Acc=65%
79.72
10.72.4
5.02
E = 5 and total client number K = 10 under 3 different training random seeds.
VHL
8R. 10
88 41
88.67
86.40
8T .9
$7.22
84:30
81.96
85. 30.
80.91
81 96
84.86
82.54
76.88
73.00
74.98
89 a:
0.45 74.95
0.85 83.8 0.30
00788
0.78
3,89
=3.0
CCVR
83.05
84:20 20
81,63
83-87
83.00
83.88
83:32
83:12
83.48
78.64
77.17
78.51
75.36
74.7u
74.50
80.87 0:31 83 84:20
81 13 78.07
78.07
1.35 70 08.98 2.74 74.85
86
50,80
0.83
13 1.09
5.52
SCAFFOLD
82 00
80.10
31.57
61.5 59
81 11.00
82.14
$0.06
73.30 13.32
72.6
74.14
60. 23
80
83
81
70
60.
77.8144
72
77.8
0.17
# 1.16 83.87 = 0.85 89.79 - 0.82
0.83
FidP
85.52
80.33 33
85,95
84.31
83.97
8435
Msant: std 81.76 # 1-48 85, 72 # 1.24 84.1=
82.87
84.03
84.46
70. 16
76.33
77-75
74.00
72.67
72.0
=
7681
73.09
80.74
85.08 0.30
22222 22 75.16 = 0.68
2. 19 69.64 1.36
80,05 6,80
Find vggg
55,00
86,28
65.75
86.84
53.98
8-8.88
844.05
88.61
82.04
75.30
74.37
75.54
70.35
70.51
68.07
86 0.37
7,09
Fedlkng
56 40
55.67 .7
$5.36
96 85
83.08
$4.21
82.07 17
62.02 022
72.71
73.22 22
00.14 14
06.51
66. 41
89.67
55,
Mcant: std 82
Meunz std 71
Mean1 std 07
78.5
Meant: std
seed 0|
seed 2
seed 0|
secd 0
Heterogeneous scomario 4 seed 1
2
sood 0|
Heltnngeeeeus scemarti 5 seed 1
seed 2
Tocal Apc std
Heterogenaaus secnario I seed
seed
Hoterogencuus sconario sood
seed
Heterogennnus secnario 2
Datoset:
41

---

90
Accuracy [%]
60
30
FedAvg
FedAvgM
FedASAM
Ours
25
50
75
100 Round
125
150
175
200
Figure 15: The training process visualization of top-5 baselines and our method FedGPS on CIFAR 10, heterogeneity degree a 0.1, local epochs E. 5 and total client number K under heterogeneous scenario 1.
42


Reviews:
[{"review_id": "lNzLkcLjQh", "reviewer": "Reviewer_2KuF", "summary": "This work conducts comprehensive evaluations across different heterogeneous scenarios, and asserts most existing methods exhibit limited robustness. Motivated by the observation, this work proposes FedGPS, which statically modifies each client's learning objective to resemble the global data distribution, while adjusting local update directions with additional gradient information.", "questions": "- Doesn’t incorporating gradient information from other clients incur additional computational cost? In Def 4.2, it seems that to compute $\\delta_{\\theta_i}^t$, each participating client has to communicate $\\Delta_{\\theta_k}^{t-1}$. Is this incorporated as \"communication round\" in the experiments? \n\n- By communicating local gradient information with other clients, can it hurt the improved privacy aspect of federated learning? \n\n- How come comparison to adaptive methods [1, 2, 3] are not included? These also try to address heterogeneity issue in federated learning.\n\n[1] S. Reddi et al., Adaptive Federated Optimization\n\n[2] J.L. Kim et al., Adaptive Federated Learning with Auto-Tuned Clients\n\n[3] S. Mukherjee et al., Locally Adaptive Federated Learning", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strength:\n- The paper is well-written and easy to follow. \n- Reported empirical performance is strong, although the reported metrics seem a bit unfair, if the communication cost of non-self gradient computation is not taken into account.\n\n### Weakness:\n- It seems that the proposed method requires many more communications than existing methods. In particular, I'm confused about the non-self gradient for each client. It seems that each client has to communicate high-dimensional vector to every other client. This is even more expensive than one \"communication round\" of federated learning in the traditional sense (every participating client periodically communicates with the server). Is this true?", "was_revised": true, "final_justification": "The authors provided very detailed answer to my concerns. In particular, they clarified that the clients do not have to share the local gradient with other clients. They further provided additional experimental results, including a variant of their proposed method with the same communication complexity with FedAvg, and still exhibit competitive performance. I am thus raising my score.", "rebuttal": "We are truly grateful for the reviewers' dedicated efforts, which greatly improve the quality of FedGPS. Their **well-written and easy to follow** comments were invaluable, and we especially appreciate their **strong affirmation of our experimental results**. Additionally, we provide the following answers to the reviewers' concerns:\n>**_Q1:_** FedGPS requires many more communications than existing methods. It seems that each client has to communicate a high-dimensional vector to every other client. Extra computational cost. Incorporating gradient information from other clients incurs additional computational cost? By communicating local gradient information with other clients, can it hurt the improved privacy aspect of federated learning?\n\n_**A1:**_ Thank you for your valuable feedback and for raising this important point regarding the communication and computational costs of FedGPS. In summary, \n- During the download stage, aligning with your valuable feedback, there is one additional model (containing the aggregated gradient) of the same size as the global model. \n- During the upload stage, **there is no additional communication overhead**. \n- During training, **no communication is required between clients**, namely, only communication is needed between the client and the server.\n- We **further implement a variant with no extra communication overhead** called FedGPS-CF, showing comparable performance with FedGPS.\n\nThus, FedGPS protects privacy as previous works and brings about 1.5 times communication overhead than vanilla methods, e.g., FedAvg.\n\nInspired by your constructive comments, we have provided a detailed table description of the two consecutive upload and download rounds in the revised version, along with the associated local computational requirements. Additionally, we include a comparison with FedAvg to demonstrate that FedGPS does not impose significant additional overhead. We denote the whole model size as $M$ and the total classes of the dataset as $C$, e.g., $C=10$ for CIFAR-10. \n\n**Table 1. The whole process comparison between FedAvg and FedGPS (Ours)**.\n|Process|FedAvg|FedGPS|\n|:-:|:-:|:-:|\n|Global aggregation at round t-1|Update global model$\\theta^{t}=\\sum\\theta_{k,E}^{t-1}$|1.Update global model $\\theta^{t}=\\theta^{t-1}+\\eta_g\\sum_{k\\in\\mathcal{S}_{t-1}}\\Delta_k^{t-1}$ 2.Update global surrogate protoypes $\\mathcal{E}^{c}=\\sum\\mathcal{E}^c_k$|\n|Explanation||Apart from the direct aggregation parameters, e.g, FedAvg-like, the $\\Delta$ of client parameters has also been widely used in many studies[R1-R4].|\n|Server|Select subset $\\mathcal{S}_t$ to participate Round t|Select subset $\\mathcal{S}_t$ to participate Round t|\n|Round t selected $\\mathcal{S}_t$ **download** from server|Global model $\\theta^{t}$ **(#Comm $M$)**|1.Global model $\\theta^{t}$;2.Global model update infomation $\\Delta \\theta^t=\\theta^{t}-\\theta^{t-1}=\\eta_g\\sum_{k\\in\\mathcal{S}_{t-1}}\\Delta_k^{t-1}$ **(#Comm $2M+C*512$)**|\n|Explanation||$\\Delta\\theta^t$ contains the gradient aggregation information updated by the selected client in the previous round $t-1$, The global surrogate distribution is represented by a prototype for each class. The prototype for each class is a 512-dimensional vector.|\n|Local opereation|Update local model $\\theta^{t}_{k,0}=\\theta^{t}$ ($0$ means the model without local epochs training)|1.Update local model $\\theta^{t}_{k,0}=\\theta^{t}$;2.Compute Non-Self Gradient based on $\\Delta\\theta^{t}$. |\n|Local extra opereation explanation|| **A:** $\\delta_{\\theta}^{t}=\\Delta \\theta^{t}-\\Delta_{k}^{t-1}$ If this client is selected last round which means we should distract its local update $\\Delta_{k}^{t-1}$ of last round (this is kept locally) **B:** $\\delta_{\\theta}^t=\\Delta \\theta^{t}$ If this client is not selected last round which means $\\Delta \\theta^{t}$ contains all other client's gradient information. |\n|Local training|Traditional SGD use loss function and local data|SGD use Eq(6) in FedGPS with local data|\n|Explanation||Incorporating gradient information from other clients only occurs by adding the parameters together before each gradient descent. This additional computation overhead is almost negligible and can be disregarded. There are a total of iteration times of parameter summation operations. (**Negligible additional computation expenses**)|\n|Round t selected $\\mathcal{S}_t$ **upload** to server|New local model parameters $\\theta^{t}_{k,E}$ **(#Comm $M$)**|1.Local updated parameters $\\Delta_{k}^{t}=\\theta_{k,E}^{t}-\\theta_{k,0}^{t}$ 2.Compute local surrogate prototypes $\\mathcal{E}_{k}^{c}=\\frac{1}{\\|\\mathcal{D}_s^c\\|}\\sum\\psi_k(\\xi_s^c)$ **(#Comm $M+C*512$)** |\n|Global aggregation at round t|$\\theta^{t+1}=\\sum\\theta_{k,E}^{t}$|$\\theta^{t+1}=\\theta^{t}+\\eta_g\\sum_{k\\in\\mathcal{S}_{t}}\\Delta_k^{t}$|\n\nThe extra $C*512$ (where $C * 512 \\ll M$，e.g., 0.05% in CIFAR-10) is the local uploaded local surrogate distribution prototypes and downloads the global surrogate distribution prototype. （The detailed explanation of local and global surrogate information can be referred to the reply to **_Reviewer 3XPZ Q1_**）.\nTo further clarify, we emphasize that in each round of FedGPS, **we do not require all clients to send their own gradients to other clients**. Instead, based on the information from the selected clients in the previous round, we aggregate their gradients for dual purposes: updating the global model and providing aggregated gradient information for client selection in the next round. As a result, the upload cost remains identical to that of vanilla FedAvg. During the download from the server, clients receive not only the latest model but also the aggregated gradient information from the server. This aggregation process inherently protects individual client gradients from being accessed by others (**addressing privacy concerns**). Consequently, throughout the entire FedGPS process, there is no direct information exchange between any clients, and all information obtained from the server is solely aggregated data.\n\nInspired by your valuable comments, **we further design a communication-friendly variant of FedGPS, called FedGPS-CF**. Specifically, when **uploading** the model, FedGPS-CF, like FedGPS, **only has a communication cost of $M+C*512$**. When **downloading** from the server, it **still only downloads the global model and global surrogate prototypes, meaning the communication cost remains $M+C*512$**. Here, we use the difference between the global model downloaded in two rounds from the server to represent the gradient aggregation information of other clients. Similarly, if a client is selected in both adjacent rounds, its own update information should be removed. Finally, FedGPS-CF achieves comparable performance to FedGPS, the results are listed in Table 2. However, it reduces the download overhead of $M$, making FedGPS-CF only have an additional communication cost of $C*512$ compared to FedAvg, which is negligible. Once again, we thank you for your help and for inspiring our paper.\n\n**Table 2. The results between FedGPS-CF and FedGPS under Heterogeneous scenario 1 with CIFAR-10**.\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedGPS-CF|90.01|88.13|78.07|\n|FedGPS|90.31|88.47|78.32|\n\n>**_Q2:_** How come comparison to adaptive methods are not included? These also try to address heterogeneity issue in federated learning.\n>\n_**A2:**_ We agree that comparing a new method with adaptive methods is necessary, since these methods are also outstanding approaches to address data heterogeneity in FL. Following your valuable comments, we have **selected two representative adaptive methods** $\\Delta$-SGD and FedAdam, from references [R5, R6] and **incorporated them as additional baselines in our experiments**. Along with the other methods already evaluated, these comparisons further demonstrate the superiority of FedGPS. The results are listed in Tables 3, 4, and 5. We have discussed these papers [R5-R7] and listed these results in our revised version.\n\n**Table 3. The results in FedGPS under Heterogeneous scenario 1 with CIFAR-10.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedAdam [R5]|85.34|85.05|67.43|\n|$\\Delta-SGD$ [R6]|86.96|86.66|69.49|\n|**FedGPS(Ours)**|**90.31**|**88.47**|**78.32**|\n\n**Table 4. The results in FedGPS under Heterogeneous scenario 1 with SVHN.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedAdam|89.80|91.01|93.27|\n|$\\Delta-SGD$|89.78|90.21|94.08|\n|**FedGPS(Ours)**|**94.20**|**93.61**|**95.03**|\n\n**Table 5. The results in FedGPS under Heterogeneous scenario 1 with CIFAR-100.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedAdam|69.89|65.33|55.43|\n|$\\Delta-SGD$|70.07|67.78|57.48|\n|**FedGPS(Ours)**|**71.14**|**68.90**|**58.77**|\n\nReferences:\\\n[R1] Durmus Alp Emre Acar, et al. Federated learning based on dynamic regularization In ICLR, 2021.\\\n[R2] Taehwan Lee, et al. Rethinking the Flat Minima Searching in Federated Learning. In ICML, 2024.\\\n[R3] Ziqing Fan, et al. Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization. In ICML, 2024.\\\n[R4] Wenjing Yan, et al. Problem-Parameter-Free Federated Learning. In ICLR 2025.\\\n[R5] Reddi, Sashank J., et al. Adaptive Federated Optimization. In ICLR, 2021.\\\n[R6] Kim, Junhyung Lyle, et al. Adaptive Federated Learning with Auto-Tuned Clients. In ICLR, 2024.\\\n[R7] Mukherjee S, et al. Locally adaptive federated learning. In TMLR, 2023.\\\n[R8] Karimireddy S P, et al. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020."}, {"review_id": "ImoPMx9XZN", "reviewer": "Reviewer_LAwd", "summary": "Federated Learning (FL) struggles with data heterogeneity, limiting model performance. Existing methods lack robustness across diverse scenarios. This work shows that sharing statistical information helps mitigate heterogeneity by providing a global perspective. Inspired by this, the authors propose ​​FedGPS (Federated Goal-Path Synergy)​​, a novel framework that combines statistical distribution and gradient information from other clients. FedGPS statically adjusts learning objectives to model global data distribution and dynamically refines local updates using external gradient information. Experiments demonstrate that FedGPS outperforms existing methods in various heterogeneity scenarios, proving its effectiveness and robustness across datasets.", "questions": "Please refer to the weakness part.", "limitations": "Please refer to the weakness part.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes a robust approach (FedGPS) that effectively integrates statistical and gradient information to address data heterogeneity.\n2. This paper conducts experiments across diverse datasets and heterogeneity scenarios.\n3. This paper is easy to read and follow.\n\n\nWeaknesses:\n1. For Figure 1, could the authors provide more explanation about the Nemenyi post-hoc test? This method appears less commonly used in evaluations.\n2. In line 79, the authors describe refining the update direction using other clients' information. Would it be possible to support this with theoretical justification?\n3. For Equation 7, could the authors clarify:\n- How is λ determined?\n- Why does incorporating non-self gradients improve the update direction?\n4. Have the authors considered evaluating their method on Vision Transformers (ViTs) or NLP tasks?", "was_revised": true, "final_justification": "The comments have addressed most of my concerns. I decide to raise my score.", "rebuttal": "We sincerely thank the reviewers for their invaluable assistance and insightful contributions to our paper. We are particularly pleased to hear that our paper was found to be **easy to follow** and that our **extensive experimental work was acknowledged**. We now proceed to address the concerns raised by the reviewers\n>**_Q1:_** Provide more explanation about the Nemenyi post-hoc test. This method appears to be less commonly used in evaluations.\n\n_**A1:**_ Thanks for pointing out this potentially confusing setting. Accordingly, we have added the following explanations in the revision. \n\nThe Nemenyi post-hoc test in Figure 1 is a non-parametric statistical method used for pairwise comparisons of multiple groups [R1] (e.g., algorithms or models) after a significant result from a Friedman test (a non-parametric analog to repeated-measures ANOVA). It ranks the performance of each method across multiple independent runs or datasets and computes a \"critical distance\" (CD) threshold. If the average rank difference between two methods exceeds the CD, their performances are considered statistically significantly different at a given significance level (typically α=0.05). The test is conservative and accounts for multiple comparisons to control the family-wise error rate, making it robust for scenarios like ours, where we evaluate algorithm robustness across diverse heterogeneity partitions (e.g., different random seeds for Dirichlet distributions). \\\nIn Figure 1, the Nemenyi post-hoc test assesses the robustness of baseline methods across different heterogeneity scenarios. The results show overlapping CD intervals for most baselines, indicating no statistically significant performance differences among them. This finding highlights the need for our proposed approach, as existing methods exhibit limited adaptability to varied data distributions. Furthermore, the **Nemenyi test has been widely adopted in holistic evaluations [R2-R5] and federated learning scenarios[R6-R7]**.\n\n>**_Q2:_** In line 79, the authors describe refining the update direction using other clients' information. Would it be possible to support this with theoretical justification?\n\n_**A2:**_  Thanks to the reviewers' valuable comments to improve the quality of our paper. The **insight behind incorporating information from other clients in FedGPS stems from works like SCAFFOLD** [R8] and other related research [R9], which also use other client information as a control variate to adjust update direction. Beyond this intuition, we provide a theoretical justification using a Taylor expansion to demonstrate that integrating other clients' information can indeed **further decrease the deviation between local and global update directions**. Here's a detailed proof:\n\nWe denote local loss function as $f_k(\\cdot):\\mathbb{R}^d\\rightarrow\\mathbb{R}$ and global loss function $F(\\cdot)$. First of all: The original local update uses the vanilla gradient descent on the local model $\\theta_k$ is denoted as $g_{\\text{old}}=\\nabla f_k(\\theta_k).$ For FedGPS, we incorporate non-self gradient $\\delta_{\\theta_k}$ to local model , we denote $\\frac{\\delta_{\\theta_k}}{||\\delta_{\\theta_k}||}$ as $g_k'$:\n$$\\theta'_k=\\theta+\\lambda_g g_k'.$$\nThen we get a new update direction computed based on the new model parameters $\\theta_k'$:\n\n$$g_{\\text{new}}=\\nabla f_k(\\theta_k').$$ \nFor a continuously twice-differentiable function $f_k(\\theta)$, the gradient function$\\nabla f_k(\\theta)$ expands around point $\\theta$ along direction $g_k'$:\n$$\\nabla f_k (\\theta_k+\\lambda_g g_k')\\approx \\nabla f_k(\\theta_k) +\\nabla^2 f_k(\\theta_k)(\\lambda_g g_k')+R_3,$$\nwhere $R_3$ represents higher-order terms that can be neglected, and $\\nabla^2 f_k$ is the Hessian at $\\theta_k$. Thus, we can get:\n$$\\nabla f_k (\\theta_k')\\approx \\nabla f_k(\\theta_k) +\\lambda_g \\nabla^2 f_k(\\theta_k)g_k' .$$\nHere we assume that the loss function is convex. So the Hessian $\\bar{H}=\\nabla^2 f_k$  is positive semi-definite. Ideally, we assume non-self gradients contain all the gradients from other clients; we can denote $\\delta_{\\theta_k} = \\nabla F(\\theta) - \\nabla f_k(\\theta_k)$. Substitute into the expansion:\n$$\\nabla f_k(\\theta_k') \\approx \\nabla f_k(\\theta_k) + \\lambda_g \\bar{H} (\\nabla F(\\theta) - \\nabla f_i(\\theta))=(I - \\lambda_g \\bar{H}) \\nabla f_k(\\theta_k) + \\lambda_g \\bar{H} \\nabla F(\\theta),$$\nwhere $I$ is the identity matrix. As a result:\n- The original bias between local and global gradient: $d_0 = ||g_{\\text{old}} - \\nabla F(\\theta)||$\n- Refined update direction bias between new model parameters $\\theta_k'$ and global gradient: $d'= \\|g_{\\text{new}} - \\nabla F(\\theta)\\| \\approx \\|(I - \\lambda_g \\bar{H}) (\\nabla f_k(\\theta) - \\nabla F(\\theta))\\| \\leq \\|I - \\lambda_g \\bar{H}\\| \\cdot d_0$. If $\\lambda_g$ is tuned to make $||I - \\lambda_g \\bar{H} < 1||$, then $d'<d_0$, which reduces the shift.\n\nIn practice, direct access to all client gradient information is often limited due to privacy and communication overhead. Nevertheless, through careful hyperparameter tuning, FedGPS consistently achieves state-of-the-art (SOTA) performance across various heterogeneous scenarios.\n\n\n>**_Q3:_** For Equation 7, could the authors clarify:(1)How is $\\lambda$ determined?(2)Why does incorporating non-self gradients improve the update direction?\n\n_**A3:**_ Thanks to the reviewers for their valuable questions. \n- Our approach is **not very sensitive to different $\\lambda_g$**, we **conduct the grid search** for $\\lambda_g\\in${$0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0$}. The results are listed in the following Table 1. Therefore, in order to save the time of hyperparameter tuning, we choose 0.5 as a fixed parameter. The following results in our paper also prove that this parameter can still make FedGPS achieve the effect of SOTA. This also shows that **FedGPS does not need to spend too much time on parameter tuning.**\n\n**Table 1. Different hyperparameters of $\\lambda_g$.** \n||0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1.0|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|FedGPS|77.55|77.68|77.99|77.37|**78.32**|77.88|78.08|77.93|77.79|77.87|\n\n- Regarding why incorporating non-self gradients improves the update direction, our **intuition stems from works like SCAFFOLD[R8], which show that integrating information from other clients can refine one's own update direction**. Furthermore, we also provide a theoretical justification that incorporating the information from other clients can further decrease the deviation between local and global update directions. The detailed proof is listed in the response to **_Q2_**.\n\nWe are grateful for the reviewer's constructive comments, which have enabled us to offer a more thorough explanation of FedGPS's SOTA performance.\n\n>**_Q4:_** Have the authors considered evaluating their method on Vision Transformers (ViTs) or NLP tasks?\n\n_**A4:**_ We sincerely appreciate your constructive question. Accordingly, we explore the applicability of our method beyond the current evaluations. The following results and discussions have been added to the revision.\nTo verify the effectiveness of our method, **we further incorporate the ViT model in [R10] for FL**. The results, which demonstrate consistent improvements in accuracy compared to baselines, have been reported in Table 2.\n\n**Table 2.ViT-based model under CIFAR-10 dataset with $\\alpha=0.1$.**\n|| K=100,$\\lambda_s$=10%,R=500, E=1|K=10,$\\lambda_s$=50%,R=500, E=1|\n|:-:|:-:|:-:|\n|FedAvg|30.45|60.85|\n|FedAvgM|33.56|60.97|\n|FedProx|48.46|59.32|\n|SCAFFOLD|35.38|63.56|\n|CCVR|-|61.92|\n|VHL|47.36|70.09|\n|FedASAM|35.43|58.55|\n|FedExp|28.35|-|\n|FedDecorr|39.63|61.18|\n|FedDisco|-|59.19|\n|FedInit|32.34|39.98|\n|FedLESAM|45.83|63.69|\n|NUCFL|38.89|62.33|\n|**FedGPS(Ours)**|**56.71**|**73.56**|\n\nDue to limited time, we will incorporate more experimental results of ViT in various scenarios in the upcoming version.\n\nRegarding NLP tasks, our method may require additional adaptations to effectively handle sequential data in NLP, especially how to design surrogate information for statistical information alignment. While we have not yet conducted these experiments due to the need for further methodological refinements, this insightful question has inspired us to prioritize NLP evaluations in future extensions of our work. We appreciate the reviewer's insightful comments in broadening the scope of our research.\n\nReferences:\\\n[R1] Nemenyi, Peter Bjorn. Distribution-free multiple comparisons. Princeton University, 1963.\\\n[R2] Demšar, Janez. Statistical comparisons of classifiers over multiple data sets. In JMLR 2006.\\\n[R3] Chen, Xiangzhi, et al. Disentangling cognitive diagnosis with limited exercise labels. In NeurIPS, 2023.\\\n[R4] Jansen, Christoph, et al. Statistical multicriteria benchmarking via the GSD-front. In NeurIPS, 2024.\\\n[R5] Guidotti, Riccardo, et al. Generative model for decision trees. In AAAI, 2024.\\\n[R6] Yang, Yuwen, et al. Federated multi-task learning on non-iid data silos: An experimental study. In ICMR, 2024.\\\n[R7] Song, Yukun, et al. Causal multi-label feature selection in federated setting. In arXiv:2403.06419, 2024.\\\n[R8] Karimireddy S P, et al. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020.\\\n[R9] Wenjing Yan, et al. Problem-Parameter-Free Federated Learning. In ICLR 2025.\\\n[R10] vision-transformers-cifar10: Training Vision Transformers (ViT) and related models on CIFAR-10."}, {"review_id": "K7pPShjkCx", "reviewer": "Reviewer_3XPZ", "summary": "The paper proposed FedGPS, a framework that integrates statistical distribution and gradient information from other clients in federated learning, aiming to mitigate data heterogeneity issues and improve the performance of trained models. The framework consists of a static goal-oriented objective that incorporates feature alignment between local real and surrogate distributions, as well as alignment between local and global surrogate distributions, and dynamic path-oriented rectification that rectifies local update direction using gradients from other clients. Experiments showed that the proposed method achieved improved accuracies and convergence speed compared to previous work.", "questions": "1. Why is it necessary to introduce the surrogate distributions? Based on the assumption that the global surrogate distribution is computed from the global real distribution (which is not defined in the paper), if equation (6) is the objective function for local update, then why not simply use the real global and local statistics, i.e., use the distance between the local feature and global feature (sent from the server) in the loss function? In addition, what is purpose of aligning the local real distribution and the local surrogate distribution if the latter stems from the former one?\n\n2. What is the communication overhead introduced by the extra statistical and gradient information transmission?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposed an effective method to tackle data heterogeneity issues. Statistical information when properly used can help alleviate the divergence between local and global models while preserving user privacy. The proposed method is intuitive and could be applied in a wide range of cases.\n\n2. The paper conducted extensive experiments to show the performance of the proposed method, which improves over previous work in most cases.\n\nWeaknesses:\n\n1. The paper did not formally define the local surrogate distribution and global surrogate distribution, which are crucial components of the proposed method. In addition, more information needs to be transmitted between the server and clients such as the global features and gradients of other clients. It should be made clear what information is transmitted in each round of communication.\n\n2. The notations are inconsistent in Sections 3 and 4. Below are some examples:\n\n- In line 111 the set of clients is K but in line 112 it is S.\n- In line 163, sum of distributions.\n- In lines 175-176, Stage 1 and Stage 2 are not further defined or referenced.\n- In line 180, \\delta in bound.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns.", "rebuttal": "We deeply appreciate the reviewer's efforts, particularly their recognition that the proposed method is **intuitive and could be applied in a wide range of cases**, as well as their **affirmation of our experimental results**. We also address the questions raised in the following.\n>**_Q1:_** Formally define the local surrogate distribution and global surrogate distribution. \n\n_**A1:**_ Thank the reviewers for helping to make our paper clearer and easier to understand. In summary,\n- We **give the formal definition** of local and global surrogate distribution.\n- The detailed process of **how to use local and global surrogate distribution of FedGPS in practice**.\n\nWe first recap some notations in Table 1(we will revise some notations in the next version). Then we give the formal define of local surrogate distribution and global surrogate distribution in the following:\n\n**Table 1. Some notations.**\n|Notation|Meaning|\n|:-:|:-:|\n|$\\mathcal{D}_s$| Surrogate Dataset(No raw information, **all clients have the same dataset**)|\n|$\\mathcal{D}_k$| Local raw dataset, each client hold its own|\n|$\\psi_k$|Feature extractor of client k|\n|$\\xi_s$| Data sample from surrogate dataset|\n|$\\xi_k$|Data sample from local dataset of client $k$ |\n|$c$| $c$-th class in dataset|\n|$\\mathcal{E}^c$|Embedding representation of $c$-th calss (512-dimension in our paper)|\n\n**Definition 1.(Local Surrogate Distribution).** For a client $k$ in a federated learning system, the **Local Surrogate Distribution ($\\mathcal{P}_{k}^{s}$)** is conceptually defined as the set of feature embeddings obtained by passing each data point from surrogate dataset through the $k$-th local model's feature extractor($\\psi_k$).\n\n**Definition 2.(Global Surrogate Distribution).** At the server side, the **Global Surrogate Distribution ($\\mathcal{P}^{s}$)** is defined as the set of feature embeddings obtained by passing each data point from surrogate dataset through the global extractor($\\psi$).\n\nIn the implementation, to ensure privacy and reduce communication overhead, what is transmitted to the server is a compressed, privacy-preserving statistical representation of surrogate distribution. Typically these embeddings are then aggregated to form a set of class-wise prototype vectors (e.g., 512-dimensional), with each prototype representing a specific class. This distribution serves as a compact proxy for local surrogate distribution:\n$$\\mathcal{E}_{k}^{c}=\\frac{1}{|\\mathcal{D}_s^c|}\\sum\\psi_k(\\xi_s^c),\\tag{1}$$\n\nwhere $ \\mathcal{D}^c_s $ denotes the subset of the surrogate dataset belonging to class $c$. Follow the Eq(1) we can get the $C$ prototypes for each class in the surrogate dataset to present the local surrogate distribution in a privacy-preserving and communication-friendly way. The global surrogate distribution is also replaced with aggregated local surrogate prototypes (only selected clients').\n$$\\mathcal{E}^{c}=\\sum_{k\\in\\mathcal{S}_t}\\mathcal{E}_k^c\\tag{2} $$\nThis prototype-based method also been applied in many FL works e.g.,[R1-R4]. **Moreover, FedGPS only transmit the the aggregated surrogate data's prototype, which is safer than the local data's prototype.** \n\n\n>**_Q2:_** More information needs to be transmitted between the server and clients such as the global features and gradients of other clients. It should be made clear what information is transmitted in each round of communication.\n\n_**A2:**_ We sincerely appreciate your kind suggestion. Due to the limited rebuttal space, the detailed communication overheads are listed in Table 1 of the response to **_Reviwer 2KuF_**. In summary, \n- Our method **only introduces additional $C*512$** communication overheads during the upload process, which is much smaller than the original model. \n- During the download process, in addition to the global model, **an additional aggregated gradient information from the previous round of participating clients is required.** This is the same size as the original model, and there is also global aggregated global surrogate prototypes (this can still be ignored). Therefore, **the additional communication overhead of FedGPS is limited.**\n- Inspired by the comments, we **implement a communication-friendly version of FedGPS**, called FedGPS-CF. Compared to FedAvg, **only additional class-wise prototypes that can be ignored are required** still show comparable peroformance with FedGPS.\n\n>**_Q3:_** Notation problems:\n\n_**A3:**_ We appreciate your careful review and highlighting these notation inconsistencies in Sections 3 and 4. We will revise the text accordingly in the next version.\n- In line 111 the set of clients is K but in line 112 it is S.\n    - We will fix this problem in the next version to replace $\\mathcal{S}$ with $\\mathcal{K}$ in Line 111.\n- In line 163, sum of distributions.\n    - We will replace these notations according to the response to **Q1**.\n- In lines 175-176, Stage 1 and Stage 2 are not further defined or referenced.\n    - Stage 1 denotes the alignment at local period in the Fig.3(a) and Stage 2 is the alignment at local global stage in Fig.3(a). We will fix these problems. \n- In line 180, \\delta in bound.\n    - We will fix this problem in the next version to replace $\\delta$ with $\\kappa$ in the Theorem 4.1.\n\n>**_Q4:_** Why is it necessary to introduce the surrogate distributions? Based on the assumption that the global surrogate distribution is computed from the global real distribution (which is not defined in the paper), if equation (6) is the objective function for local update, then why not simply use the real global and local statistics, i.e., use the distance between the local feature and global feature (sent from the server) in the loss function? In addition, what is purpose of aligning the local real distribution and the local surrogate distribution if the latter stems from the former one?\n\n_**A4:**_ The surrogate dataset is sampled from different Gaussian distributions representing distinct classes. Thus, the surrogate dataset is different from local datasets (**without any information about raw data**). **Secondly,** extensive experiments also demonstrate the potential of sharing statistical information for alleviating data heterogeneity. Therefore, from a distribution perspective, we use a two-stage alignment method to align the locally biased statistical information with the global data while protecting privacy. **Lastly,** FedGPS employs a two-stage approach. On one hand, it **prevents the directly exposure of the statistical information of the original data**. On the other hand, all clients use the same surrogate distribution, which reduces the deviation of both global and local statistical information. (The gap can be bridged by aligning the distribution of different models on the same pure noise dataset)\n\nReferences:\\\n[R1] Huang, Wenke, et al. Rethinking Federated Learning with Domain Shift: A Prototype View. In CVPR, 2023.\\\n[R2] Tan, Yue, et al. Fedproto: Federated prototype learning across heterogeneous clients. In AAAI, 2021.\\\n[R3] Dai, Yutong, et al. Tackling data heterogeneity in federated learning with class prototypes. In AAAI, 2023\\\n[R4] Wang, Lei, et al. Taming cross-domain representation variance in federated prototype learning with heterogeneous data domains. In NeurIPS, 2024."}, {"review_id": "5qL5V092xJ", "reviewer": "Reviewer_V82G", "summary": "This paper first identified a key issue that many existing FL algorithms lack robustness across diverse data heterogeneity scenarios. To address this problem, Federated Goal-Path Synergy (FedGPS) is proposed, aiming to enhance the robustness by leveraging both statistical and gradient information from other clients. Specifically, the author proposed a static goal-oriented objective function, including two stages: (1) aligning local distributions with a local surrogate distribution, and (2) aligning the local surrogate distribution with the global surrogate distribution. Another technique, dynamic path-oriented gradient rectification, is proposed to integrate non-self gradient information into local updates to correct the local optimization directions.", "questions": "1. Could the authors provide additional technical details regarding the generation process of surrogate distributions? Moreover, it would be beneficial to provide some justifications for why Gaussian distribution was chosen.\n2. How do the authors plan to address potential privacy concerns associated with the proposed two techniques (i.e., surrogate distributions and share of non-self gradient information)?\n3. Have the authors considered evaluating other non-iid scenarios beyond the Dirichlet partitioning method, such as assigning each client data from exactly C distinct classes (C=2 or C=3), ensuring that each client only sees a specific subset of all labels?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper is well-motivated and clearly structured. \n- The idea of modeling the global data distribution using a two-stage surrogate distribution alignment is novel and intuitive.\n- The authors conducted extensive and thorough experiments under a wide range of non-IID conditions to demonstrate the robustness of their proposed FedGPS compared to comprehensive baseline approaches.\n\nWeaknesses:\n- I am missing some details on how exactly the surrogate distribution is generated. The paper only provides a high-level description: “sampled from distinct Gaussian distributions”.\n- The paper lacks sufficient discussion on any potential privacy risks from their proposed two techniques. The authors claim that the surrogate distribution is a privacy-preserving approach since no raw statistical information is shared. However, the local surrogate distributions might still be vulnerable to reconstruction or membership inference attacks.  Additionally, the dynamic path-oriented gradient rectification explicitly utilizes gradient information derived from other clients. More comprehensive analysis and clarification regarding the potential privacy risk associated with this gradient-sharing technique are needed.\n- The motivation provided in Section 4.3, specifically the statement, \"This insight builds on the ingenious concept that a malicious client in FL …, offering extra advantages,\" seems problematic and may require further clarification.", "was_revised": true, "final_justification": "I think the authors have clarified most of my questions, particularly those related to the scope of the experimental evaluation and privacy considerations. Accordingly, I have decided to raise my score to 4.", "rebuttal": "We sincerely thank the reviewer for the feedback, noting our paper is '**well-motivated and clearly structured**', as well as '**novel and intuitive**'. We deeply appreciate the reviewer's valuable efforts in improving the quality of our paper, and we address the concerns as follows:\n>**_Q1:_**  How exactly the surrogate distribution is generated? It would be beneficial to provide some justifications for why Gaussian distribution was chosen.\n\n_**A1:**_ We thank the reviewers for their constructive suggestions, which have significantly enhanced FedGPS's readability. Our method **assigns a distinct Gaussian distribution to each class** in the original dataset. The surrogate data is then generated by sampling from these class-specific Gaussian distributions, with each client sampling from the same distribution for a given class (Note that, unlike heterogeneous, where clients only have partial classes, for the surrogate dataset, each client has surrogate data for all classes). The **underlying insight is that any real data distribution can be effectively transformed from a Gaussian distribution** [R1]. This approach also benefits from the ease of sampling Gaussian noise, making it well-suited for resource-constrained environments like FL. Furthermore, we evaluate other data distributions, such as the Laplace distribution, and as shown in Table 1, they yielded comparable performance to the Gaussian distribution. Ultimately, we select Gaussian noise for FedGPS.\n\n**Table 1. Performance comparison of Gaussian and Laplace distribution sampling for surrogate dataset generation.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|Laplace distrbuiton for FedGPS|90.21|87.87|78.12|\n|Gaussian distrbuiton for FedGPS|**90.31**|**88.47**|**78.32**|\n\n>**_Q2:_**  Lacks sufficient discussion on any potential privacy risks from their proposed two techniques. The authors claim that the surrogate distribution is a privacy-preserving approach since no raw statistical information is shared. However, the local surrogate distributions might still be vulnerable to reconstruction or membership inference attacks. Additionally, the dynamic path-oriented gradient rectification explicitly utilizes gradient information derived from other clients. More comprehensive analysis and clarification regarding the potential privacy risk associated with this gradient-sharing technique are needed.\n\n_**A2:**_ Thank the reviewers for their comments on the paper, which have helped us improve the quality of the paper. In summary, **the two technologies of FedGPS do not transmit any additional client's information to any other client compared to traditional methods**. On the contrary, **FedGPS replaces raw data prototypes with surrogate prototypes instead, thereby further protecting privacy compared to previous works [R2-R5]**. We list the privacy clarification among surrogate distribution and gradient information in the following:\n- **Regarding the surrogate distribution privacy issue**, because the surrogate is sampled from different Gaussian distributions, **it does not contain any information related to the local data**. Further, we transmit the aggregated class-wise prototypes of surrogate data. After **aggregating all the high-dimensional embeddings of each class**, the surrogate data information is further protected. Many papers[R2-R5] also transmit using the original data prototypes, which offer a weaker level of information protection than what we have. (Detailed information can be referred to the response to **_Reviewer 3XPZ Q1_**).\n- **Regarding the gradient information privacy issue**, FedGPS **doesn't transmit the gradient information of a certain client to any other client.** Every client **only uploads its own information to the server and downloads the aggregated information from the server**. This process is the same as most other federated methods in uploading and downloading the aggregated information. (Detailed information can be referred to the response to **_Reviewer 2KuF Q1_**).\n\nWe also add these discussions to the next version. Thanks for your suggestions. \n\n\n>**_Q3:_** \"This insight builds on the ingenious concept that a malicious client in FL …, offering extra advantages,\" seems problematic and may require further clarification.\n\n_**A3:**_ Thanks to the reviewers for suggesting where misunderstandings may have occurred. Our motivation for incorporating the non-self gradient indeed **draws a high-level concept from the model replacement strategy in the backdoor to attack federated models.** In this scenario, the malicious client leverages a profound understanding of the aggregation process and the collective behavior of benign clients. By accurately anticipating **how other clients' updates contribute to the global model**, the attacker strategically crafts and scales their own malicious update. This sophisticated manipulation allows them to effectively \"replace\" the global model with their backdoored version, despite the presence of numerous benign participants. The core insight here is that **knowledge of the aggregate influence of other clients provides a significant advantage in shaping the global model**. We thank the reviewer again for pointing out this confusing part. We will clarify these insights in the next version.\n\n>_**Q4:**_ Have the authors considered evaluating other non-iid scenarios beyond the Dirichlet partitioning method, such as assigning each client data from exactly C distinct classes (C=2 or C=3), ensuring that each client only sees a specific subset of all labels?\n\n_**A4:**_ Thanks to the reviewers for their suggestions, we consider more heterogeneous partitioning methods to better show that FedGPS is applicable to other partitions besides Dirichlet. We conduct experiments where each client's data only has C classes. We test both (C=2 and C=3), the results are listed in Table 2. \n\n**Table 2. CIFAR-10 results with $\\alpha=0.1, K=100, \\lambda_s=10\\%, R=500, E=1$.**\n|| C=2|C=3|\n|:-:|:-:|:-:|\n|FedAvg|51.75|69.97|\n|FedAvgM|50.61|73.21|\n|FedProx|49.64|68.96|\n|SCAFFOLD|56.08|74.34|\n|CCVR|55.83|73.69|\n|VHL|73.53|84.16|\n|FedASAM|55.33|69.37|\n|FedExp|47.53|67.29|\n|FedDecorr|69.36|79.60|\n|FedDisco|-|-|\n|FedInit|54.80|71.12|\n|FedLESAM|66.62|79.42|\n|NUCFL|50.03|72.89|\n|**FedGPS(Ours)**|**78.17**|**85.71**|\n\nReferences:\\\n[R1] Rezende D, Mohamed S. Variational inference with normalizing flows. In ICML, 2015.\\\n[R2] Huang, Wenke, et al. Rethinking Federated Learning with Domain Shift: A Prototype View. In CVPR, 2023.\\\n[R3] Tan, Yue, et al. Fedproto: Federated prototype learning across heterogeneous clients. In AAAI, 2021.\\\n[R4] Dai, Yutong, et al. Tackling data heterogeneity in federated learning with class prototypes. In AAAI, 2023\\\n[R5] Wang, Lei, et al. Taming cross-domain representation variance in federated prototype learning with heterogeneous data domains. In NeurIPS, 2024."}, {"review_id": "FTjwGi30Hs", "reviewer": "Reviewer_uV3Z", "summary": "This paper presents a federated learning framework, named FedGPS, to address the data heterogeneous problems in FL, which 1) uses surrogate information to guide the local models to learn data distribution aligned with the global distribution and 2) refines the update direction at each step based on other client information at the gradient view.", "questions": "Please see the weakness.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n\n* S1: The problem solved in this paper, data heterogeneity in federated learning, is an important and widely concerned issue.\n\n* S2: The paper designs the method from the perspective of loss-landscape, which has good interpretability.\n\n### Weakness\n\n* W1. The paper lacks novelty and is incremental. The method proposed in the paper seems to be a combination of the method proposed in [14,23] and the method proposed in [24].\n\n* W2. Related work is incomplete and lacks some key SOTA works. For example. lack of comparison with methods based on heuristic search [R1][R2]. The motivation of these methods is similar to SAM, aiming to improve the generalization of FL by guiding training towards flat areas. Since these methods use completely different technical routes from SAM, it is incomplete for the author to only mention SAM in related work, and these methods need to be discussed and compared. In addition, discussing more the SOTA SAM-based works [R3][R4] can improve the quality of the paper.\n\n* W3. From the training curves (Figures 4, 7, 10, and 11) provided by the author, the performance improvement seems limited. In addition, the author uses the highest accuracy as the evaluation standard of accuracy in the table, but it is difficult to accurately stop the training at the round with the highest performance in the real scene. I suggest that the author use the mean and standard deviation to show the experimental results.\n\n* W4. The authors seem to have only evaluated extreme heterogeneous scenarios, not normal heterogeneous scenarios, such as \\alpha=0.3, 0.6, and IID, which are commonly used settings.\n\n* W5. The author's experiments on cross-device scenarios are insufficient, lacking experiments on more clients (e.g., K=500, 1000) and different participation ratios (the author only evaluates 10% of devices participating in local training).\n\n* W6. It is unreasonable for the author to set E=1 for the main experiments, which will cause the FL communication overhead to increase sharply. In addition, according to the experimental results in the appendix, when E=5, the advantage of the method decreases significantly, and the author needs to give an explanation.\n\n[R1] Is aggregation the only choice? Federated learning via layer-wise model recombination. KDD 2024\n\n[R2] FedCross: Towards accurate federated learning via multi-model cross-aggregation. ICDE 2024\n\n[R3] Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning. CVPR 2025\n\n[R4] Flexible Sharpness-Aware Personalized Federated Learning. AAAI 2025", "was_revised": true, "final_justification": "The author addresses the main concerns. I decicde to raise my rating", "rebuttal": "We are grateful to the reviewers for their diligent efforts and for acknowledging both the **significance of our research questions and the interpretability** of our methodology.\n>**_Q1:_** Lacks novelty and is incremental.\n\n_**A1:**_ Thank the reviewers for the questions they raised, which gave us the opportunity to further clarify the contributions of FedGPS. Extensive experiments have shown that sharing statistical information has great potential to alleviate data heterogeneity, which also inspires FedGPS. Furthermore, FedGPS **takes into account statistical information from both the distribution and gradient perspectives** to enhance a global view.  Specifically, the **shift at the distribution level** is corrected through two-stage alignment based on the surrogate dataset. At the gradient level, **the gradient information of other clients** is considered before local training from a more global perspective. FedGPS has covered many heterogeneous scenarios across various datasets, further demonstrating its robustness.\n\n>**_Q2:_** Related work is incomplete and lacks some key SOTA works. \n\n_**A2:**_ We thank the reviewers for pointing out the shortcomings of related work, and we will add a discussion of the latest paper in the next version to improve our related work[R1-R4]. Additionally, we add FedMR[R1] as our new baselines; the results are listed in Tables 1-6. We will also add the results of this heuristic search method in the next version.\n\n**Table 1.The results in FedGPS under Heterogeneous scenario 1 with CIFAR-10, the best accuracy is reported.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedMR [R1]|84.28|86.83|-|\n|**FedGPS(Ours)**|**90.31**|**88.47**|**78.32**|\n\n**Table 2. The results in FedGPS under Heterogeneous scenario 1 with CIFAR-10, average, and std accuracy are reported.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedMR|78.63$\\pm$3.20|80.33$\\pm$3.76|-|\n|**FedGPS(Ours)**|**83.93$\\pm$4.41**|**82.08$\\pm$4.96**|**68.09$\\pm$6.51**|\n\n**Table 3. The results in FedGPS under Heterogeneous scenario 1 with SVHN, the best accuracy is reported.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedMR|91.32|86.64|-|\n|**FedGPS(Ours)**|**94.20**|**93.61**|**95.03**|\n\n**Table 4. The results in FedGPS under Heterogeneous scenario 1 with SVHN, average, and std accuracy are reported.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedMR|80.01$\\pm$6.08|75.21$\\pm$5.75|-|\n|**FedGPS(Ours)**|**86.22$\\pm$6.36**|**88.91$\\pm$3.42**|**93.97$\\pm$1.02**|\n\n**Table 5.The results in FedGPS under Heterogeneous scenario 1 with CIFAR-100, the best accuracy is reported.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedMR|69.45|67.45|-|\n|**FedGPS(Ours)**|**71.14**|**68.90**|**58.77**|\n\n**Table 6. The results in FedGPS under Heterogeneous scenario 1 with CIFAR-100, average, and std accuracy are reported.**\n||K=10,$\\lambda_s$=50%,R=500, E=1|K=10,$\\lambda_s$=50%,R=200, E=5| K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedMR|67.35$\\pm$0.86|66.76$\\pm$1.79|-|\n|**FedGPS(Ours)**|**69.87$\\pm$0.57**|**67.36$\\pm$1.56**|**54.33$\\pm$1.28**|\n\n>**_Q3:_** The performance improvement seems limited. I suggest that the author use the mean and standard deviation to show the experimental results.\n\n_**A3:**_ Thanks to the reviewers for their suggestions, we show part of the mean and standard deviation results in Table 7. Based on the mean and std metrics, it can be shown that FedGPS can **achieve higher performance gain on mean** compared to the highest accuracy. More results will be updated in the next version. \n**Table 7. The mean and standard deviation results under CIFAR-10 of the original results in our paper .**\n|| K=100,$\\lambda_s$=10%,R=500, E=1|K=10,$\\lambda_s$=50%,R=500, E=1|\n|:-:|:-:|:-:|\n|FedAvg|34.26$\\pm$10.27|72.56$\\pm$8.35|\n|FedAvgM|39.72$\\pm$11.45|72.44$\\pm$8.81|\n|FedProx|35.52$\\pm$9.96|79.24$\\pm$4.17|\n|SCAFFOLD|55.26$\\pm$2.57|73.59$\\pm$6.39|\n|CCVR|61.77$\\pm$1.44|81.78$\\pm$1.34|\n|VHL|56.36$\\pm$11.03|81.61$\\pm$5.03|\n|FedASAM|30.63$\\pm$9.55|73.48$\\pm$6.86|\n|FedExp|25.59$\\pm$6.83|71.67$\\pm$6.45|\n|FedDecorr|41.84$\\pm$12.42|73.03$\\pm$6.47|\n|FedDisco|-|-|\n|FedInit|52.26$\\pm$10.5|72.32$\\pm$8.01|\n|FedLESAM|61.48$\\pm$6.18|78.45$\\pm$5.25|\n|NUCFL|39.29$\\pm$8.44|72.54$\\pm$6.78|\n|**FedGPS(Ours)**|**68.09$\\pm$6.51**|**83.93$\\pm$4.41**|\n\n\n>**_Q4:_** The authors seem to have only evaluated extreme heterogeneous scenarios, not normal heterogeneous scenarios, such as $\\alpha$=0.3, 0.6, and IID, which are commonly used settings.\n\n_**A4:**_ We sincerely appreciate the reviewers' constructive feedback, which has significantly advanced FedGPS's applicability in federated scenarios, allowing it to **cover settings from homogeneous to extremely heterogeneous**. To demonstrate this, we present the experimental results for FedGPS on the CIFAR-10 dataset, with K=100 and E=1, $\\lambda_s=10%$ in Table 8.\n\n**Table 8. The results of different federated scenarios under CIFAR-10.**\n|| $\\alpha$=0.3 K=100,$\\lambda_s$=10%,R=500, E=1| $\\alpha$=0.6 K=100,$\\lambda_s$=10%,R=500, E=1| I.I.D K=100,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|:-:|:-:|\n|FedAvg|82.67|88.07|90.67|\n|FedAvgM|84.74|88.22|91.08\n|FedProx|82.07|86.40|90.75\n|SCAFFOLD|77.41|81.94|85.81\n|CCVR|83.87|85.78|90.31\n|VHL|85.89|88.19|90.74\n|FedASAM|83.59|86.49|91.03\n|FedExp|82.71|85.47|89.33\n|FedDecorr|86.33|88.39|90.45\n|FedDisco|-|-|-|\n|FedInit|86.61|89.30|91.01\n|FedLESAM|86.44|88.46|90.43\n|NUCFL|77.56|81.65|84.94\n|**FedGPS(Ours)**|**87.60**|**89.43**|**91.22**\n\n>**_Q5:_** Lacking experiments on more clients (e.g., K=500, 1000) and different participation ratios (the author only evaluates 10% of devices participating in local training).\n\n_**A5:**_ Thanks to the reviewers for motivating a more comprehensive evaluation of FedGPS, we conduct more clients (K=500) in Table 9 to demonstrate that FedGPS still surpasses other baselines. \n\n**Table 9. More number of clients experimental results**\n|| K=500,$\\lambda_s$=10%,R=500, E=1|\n|:-:|:-:|\n|FedAvg|73.19|\n|FedAvgM|75.43|\n|FedProx|76.72|\n|SCAFFOLD|64.18|\n|CCVR|64.43|\n|VHL|80.58|\n|FedASAM|75.06|\n|FedExp|74.48|\n|FedDecorr|74.24|\n|FedDisco|-|\n|FedInit|73.05|\n|FedLESAM|80.67|\n|NUCFL|73.98|\n|**FedGPS(Ours)**|**82.18**|\n\nFurthermore, inspired by reviewers, we also perform experiments with different sampling rates; the results are listed in Table 10. \n\n**Table 10. Different sampling rates for K=100.**\n|| $\\alpha$=0.1 K=100,$\\lambda_s$=5%,R=500, E=1| $\\alpha$=0.1 K=100,$\\lambda_s$=10%,R=500, E=1|  $\\alpha$=0.1 K=100,$\\lambda_s$=20%,R=500, E=1| $\\alpha$=0.1 K=100,$\\lambda_s$=50%,R=500, E=1|\n|:-:|:-:|:-:|:-:|:-:|\n|FedAvg|34.59|48.22|67.38|72.45\n|FedAvgM|35.32|58.80|68.34|73.89\n|FedProx|32.89|52.84|65.25|74.24\n|SCAFFOLD|36.73|60.17|67.59|69.34\n|CCVR|57.93|64.06|74.25|78.78\n|VHL|57.14|72.70|76.58|80.78\n|FedASAM|34.78|46.35|61.84|66.48\n|FedExp|34.38|38.26|58.35|64.52\n|FedDecorr|-|63.69|77.31|80.03\n|FedDisco|-|-|-|-|\n|FedInit|30.56|71.01|76.89|77.59\n|FedLESAM|68.86|72.64|76.35|76.90\n|NUCFL|38.85|53.72|68.31|71.91\n|**FedGPS(Ours)**|**70.89**|**78.32**|**79.72**|**81.97**\n\nDue to limited space, in addition to the highest acc reported, **we will also include the results of mean and standard deviation in the revised version**, and thank the reviewers for their suggestions.\n\n>**_Q6:_**  It is unreasonable for the author to set E=1 for the main experiments, which will cause the FL communication overhead to increase sharply. In addition, according to the experimental results in the appendix, when E=5, the advantage of the method decreases significantly, and the author needs to give an explanation.\n\n_**A6:**_ Thanks to the reviewers for their suggestions on local epochs. In fact, our setting follows many papers [R4-R7] which set local epochs to 1. In addition, we also experimented with E=5, and because the number of local epochs increased, the overfitting of the client to the local data and the deviation from the global makes the larger local epochs more difficult to improve and optimize, but our method still achieves the SOTA effect in all scenarios, which demonstrates the robust performance of FedGPS.\n\nReferences:\\\n[R1] Hu M, et al. Is aggregation the only choice? federated learning via layer-wise model recombination. In KDD, 2024.\\\n[R2] Xing X, et al. Flexible Sharpness-Aware Personalized Federated Learning. In AAAI, 2025.\\\n[R3] Hu M, et al. FedCross: Towards accurate federated learning via multi-model cross-aggregation. In ICDE, 2024.\\\n[R4] Caldarola D, et al. Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning. In CVPR, 2025.\\\n[R5] Tang Z, et al. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In ICML, 2022.\\\n[R6] Yang Z, et al. Fedfed: Feature distillation against data heterogeneity in federated learning. In NeurIPS, 2023.\\\n[R7] Kim, Junhyung Lyle, et al. Adaptive Federated Learning with Auto-Tuned Clients. In ICLR, 2024."}]

Author rebuttal:

Meta-review: (a) Summary of claims and findings:

The paper proposes FedGPS, a federated learning framework designed to improve robustness under data heterogeneity. It combines two ideas: (1) a goal-oriented objective that aligns local models using surrogate statistical information in two stages (local real -> local surrogate; local surrogate -> global surrogate), and (2) a path-oriented rectification that adjusts local update directions with aggregated non-self gradient information provided by the server. The authors claim improved robustness across many heterogeneity settings and datasets. In rebuttal, they clarified that clients never communicate directly; only aggregated information is shared via the server. They also introduced a communication-friendly variant, FedGPS-CF, and added extensive experiments, including adaptive baselines, ViT models, wider heterogeneity regimes (including IID and milder non-IID), more clients (up to K=500), and participation-rate sweeps. They provided definitions for surrogate distributions and a Taylor-expansion-based justification for gradient rectification.

(b) Strengths:

- Addresses an important FL challenge with an intuitive combination of distributional alignment and gradient rectification.
- Clear writing and organization per several reviewers.
- Strong and now broader empirical evaluation after rebuttal: many datasets, heterogeneity regimes, client counts, and baselines (including adaptive methods).
- Practical communication clarification and a lighter FedGPS-CF variant.
- Formalization of surrogate distributions and a simple theoretical rationale for gradient rectification.
- Reported means and standard deviations added; ViT experiments included.

(c) Weaknesses / missing pieces:

- Novelty is partly incremental; components relate to prototypes and control-variate/gradient-adjustment ideas (e.g., SCAFFOLD-like intuitions). The main novelty is in the specific synergy and two-stage surrogate alignment.
- Privacy is discussed but not rigorously analyzed. While only aggregated info is shared, a formal threat model or empirical privacy risk assessment (e.g., attack simulations) is missing.
- Some notational inconsistencies in the original submission; authors committed to fix.
- Communication overhead, while clarified, still adds roughly 1.5x download vs FedAvg for the main variant; CF reduces this but details belong in the main text with clear accounting.
- Limited theory: the Taylor argument is heuristic; no convergence or robustness guarantees.
- NLP tasks are deferred; ViT coverage improved but remains limited.

(d) Reasons for decision:

On balance, the paper is a solid empirical contribution with a clear problem framing and a practical method that appears robust across many conditions. The rebuttal addressed the key concerns: communication misunderstanding, scope of evaluation (including adaptive baselines, larger K, different α/C-label splits, participation rates), and definitions/notation. Multiple reviewers raised their scores to borderline accept after rebuttal, and no major blocking flaw remains. The novelty is moderate and the theory is light, so I recommend poster rather than a higher track. Still, the breadth of experiments and practical relevance justify acceptance.

(e) Rebuttal impact and discussion:

- R2KuF (communication/privacy/adaptive baselines): Authors clarified no client-to-client comms; only server-shared aggregated gradients and prototypes. Provided detailed protocol and FedGPS-CF with negligible extra overhead; added adaptive baselines (FedAdam, µ-SGD) showing competitive or superior results. Reviewer’s concerns were resolved; score raised. I give substantial weight to this resolution.
LAwd (stats test, theory for rectification, lambda choice, ViT/NLP): Authors explained Nemenyi test usage, gave a Taylor-based justification, provided a λ grid showing low sensitivity, added ViT results, and discussed NLP as future work. Reviewer raised score. These points are satisfactorily addressed for an empirical paper.
- 3XPZ (definitions, comm clarity, notation): Authors added formal definitions of local/global surrogate distributions via class-wise prototypes and clarified round-by-round communication; committed to fix notation. Reviewer’s concerns addressed; they raised to borderline accept. This alleviates clarity concerns.
- V82G (novelty, baselines breadth, metrics, heterogeneity breadth, K/participation, E setting): Authors added SOTA/heuristic baselines (e.g., FedMR), reported mean±std, added normal heterogeneity and IID settings, K=500, participation sweeps, and discussed E=1 vs E=5 behavior. Reviewer acknowledged and raised score. While novelty remains moderate, empirical breadth is now convincing.
- uV3Z (privacy risks, surrogate generation, motivation phrasing, additional partitions): Authors detailed Gaussian surrogate generation (also tried Laplace), clarified privacy stance (aggregated prototypes and no direct gradient sharing), refined the wording around the “malicious client” analogy, and added C-class label partition results. Reviewer raised score. I consider privacy discussion improved but still largely qualitative; acceptable for poster.

I read the author response and the ensuing discussion. The rebuttal addressed the major objections; no external information beyond the reviews and rebuttal was used.
Decision: Accept (poster)