Rising from Ashes: Generalized Federated Learning via Dynamic Parameter Reset
Authors: Jiahao Wu
Ming Hu
Yanxin Yang
Xiaofei Xie
ZeKai Chen
Chenyu Song
Mingsong Chen
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Primary area: applications

Abstract: Although Federated Learning (FL) is promising in privacy-preserving collaborative model training, it faces low inference performance due to heterogeneous data among clients. 
Due to heterogeneous data in each client, FL training easily learns the specific overfitting features.
Existing FL methods adopt the coarse-grained average aggregation strategy, which causes the global model to easily get stuck in local optima, resulting in low generalization of the global model.
Specifically, this paper presents a novel FL framework named FedPhoenix to address this issue, which stochastically resets partial parameters to destroy some features of the global model in each round to guide the FL training to learn multiple generalized features for inference rather than specific overfitting features.
Experimental results on various well-known datasets demonstrate that compared to SOTA FL methods, FedPhoenix can achieve up to 20.73\% accuracy improvement.

Rising from Ashes: Generalized Federated Learning via Dynamic Parameter Reset
Jiahao Wu', Ming Hu²*, Yanxin Yang', Xiaofei Xie², Zekai Chen', Chenyu Song', Mingsong Chen
MoE Eng. Research Center of SW/HW Co-Design Tech. and App., East China Normal University
2School of Computing and Information Systems, Singapore Management University hu.ming, work @ gmail.com, mschen www.ei.ccm.ee...c.
Abstract
Although Federated Learning (FL. is promising for privacy-preserving collabo- rative model training, it suffers from low inference performance due to hetero- geneous client data. Due to heterogeneous data across clients, FL training cas- ily learns client-specific overfitting features. Existing FL methods adopt coarse- grained averaging, which can easily cause the global model to get stuck in lo- cal optima, leading poor generalization. Specifically, this paper presents novel FL framework, FedPhoenix, to address this issue. stochastically re- sets partial parameters in each round to destroy some features of the global model, guiding FL. training to learn multiple generalized features for inference rather than specific overfitting features. Experimental results on various well- known datasets demonstrate that compared to SOTA FL methods, FedPhoenix can achieve up to 20.73% higher accuracy. The implementation is publicly available http://iStrinng/eePeeeeeee
at btttps://githhe
Introduction
With Artificial Intelligence (AI) technologies widely used in privacy-sensitive applications, protect- ing data privacy during model training has become an urgent requirement. To address this issue, Federated Learning [1-7] as a privacy-preserving distributed machine learning paradigm has been proposed and used in various Al applications, such as Artificial Intelligence of Things (AloT) sys- tems [8-12], recommender systems [13, 14], and healthcare systems [15, 16), which enables multi- ple clients to collaboratively train a global model without exposure of their raw data. Specifically, conventional FL consists of cloud server and multiple clients. each FL training round, the cloud server sends the global model to each client. Clients use their raw data to train the received model and then upload the trained model back to the cloud server. By aggregating all uploaded models, the cloud server can generate new global model for the next round of training.
However, due clients' diverse preferences and data volumes, client data is often not Independent and Identically Distributed (non-IID), leading to the "client drifting" problem [17-20]. Specifically, different data distributions lead different optimization directions for local models. The traditional FedAvg strategy aggregates all local models to generate the global model, which may cause the global model's optimization directions deviate from the optimal direction, leading it easily get stuck in local optima. To optimize the FL training process, existing methods attempt to utilize correction terms [21, 17], knowledge distillation (KD) [22-25], model mutation [26, 27], client clustering [28-31], and multi-model searching [32-34] strategies.
MMing Hu and Mingsong Chen are corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Although these methods can improve the inference accuracy of the global model in non-IlD see- narios, they introduce additional overhead or privacy leakage risks. For example, correction term- based methods require the communication overhead for the transmission of global correction terms. Knowledge distillation-ban based methods require additional public data or incur computation overhead to generate proxy datasets. Clustering-baseer and multi-model searching-based methods cannot sup- port secure aggregation [35], which may increase the risk of privacy leakage. Therefore, how to improve the performance of FL in non-IlD scenarios without additional overheads or public data while integrating secure aggregation is a significant challenge in FL.
Typically, compared to centralized model training, each client in an FL setting has access to far less data, which increases the risk of overfitting local features. To improve the global model's inference performance, the local model should learn more general features rather than those specific to the local dataset. Unfortunately, due to limited access to local data, the cloud server cannot justify the quality of the global model. In addition, since each client cannot access other clients' data, it cannot determine which features are generalized. Intuitively, to avoid the global model overfitiig to client-specific features, FL training should encourage the model to perform inference using more features rather than a few specific ones. Intuitively, if the model inference depends on multiple features, losing destroying few of them has only minor impact on the final decision. In contrast, the model relies on just a handful of features, destroying any one of them can severely affect its decisions.
Inspired by the above intuition, this paper presents a novel FL framework, FedPhoenix, address the low training performance in FL. Like the Phoenix rising from the ashes, FedPhoenix randomly resets partial parameters of the global model to destroy certain features, generating different local models with different parameter resets for local training within communication round. In this way, a destroyed feature is unique to a particular device, retraining will be an alternative. contrast, if the feature is truly generalized, it will be restored as FL training progresses. In addition, when features with large weights are destroyed, the model will try to use other features for inference while learning new features. At this time, the weights of some generalized features will increase, while those of device-specific features will decrease. To ensure the model's 's convergence efficiency, FedPhoenix includes dynamic reset mechanism that adjusts the ratio of reset parameters FL training progresses. The main contributions of this work are summarized as follows:
We present novel FL framework, FedPhoenix, that employs heuristic parameter resetting mechanism to achieve generalized FL model training.
We propose an adaptive parameter reset strategy control the resetting ratio and range of each layer and a dynamic stabilization strategy to ensure the convergence of FedPhoenix.
We present a thcoretical analysis of FedPhoenix's 's convergence in convex settings and con- duct comprehensive experiments on well-known benchmarks to demonstrate its effective- ness and convergence in non-convex settings.
2 Preliminary and Related Work
2.1 Problem Formulation
Consider federated learning system with N clients, each holding local dataset D; = {2, } where 2, (x y, (1)) is data sample. The goal is to collaboratively learn global model w E Rdd by minimizing the objective:
min F(w)= = Σn N mi F((w). EN
i=1 n
n. E/ n Di=1 n; is the total number of samples. Here, E(-) is sample-wise loss (e.g., cross-entropy). The server coordinates iterative training: clients locally update the global model on their data, and the server aggregates these updates (e.g., via weighted averaging) to refine the global model.
where F.(w)= =
2.2
Related Work
To address non-llI) challenges in Fl., existing methods often adopt various optimization strategies that come with inherent limitations. FedProx [21] introduces regularization term to align local

---

models with the global model, reducing client drift by penalizing deviations. While this approach incurs minimal communication overhead, its accuracy improvement is often limited, especially in highly heterogeneous scenarios. ClusterSampling [28] groups clients by data or model similarity to improve aggregation representativeness. However, its computational complexity scales quadrat- ically with the number of clients, making it impractical for large-scale deployments. FedGen [23] leverages knowledge distillation with synthetic proxy data to achieve knowledge transfer among clients. FedMut [26] injects diversity by mutating the global model through gradient perturbations. While this method demonstrates some effectiveness exploring the solution space, its reliance on random mutations can lead to inconsistent updates, potentially destabilizing convergence. In addi- tion, most existing methods inevitably incur additional communication overhead or require public data, cannot be employed in secure aggregation, which significantly limits generalization.
To the best of our knowledge, FedPhoenix the first FL. method to use the heuristic parameter reset strategy to enhance generalization without additional data or communication overhead, and it compatible with the secure aggregation strategy.
Prestudy and Our Motivation
Intuition. In non-IID scenarios, the limited diversity and volume of local data often cause the global model to overfit specific local features. Intuitively, the model inference process is more focused on few overfitting features rather than on multiple generalized features, resulting in low inference performance for the global model trained by FedAvg.
Conrestatiol Foard
aiioiin Beands
(a) Acc. (DTrain)
(b) Loss (Drrain)
(c) Acc. (Drest)
(d) Loss (Drest)
Figure 1: Accuracy and loss curves of FedAvg and FedPhoenix with a = 0.6
Study the Overfitting Issue of the Global Model. To explore the overfitting issue of FL, conducted a prestudy for FedAvg on the CIFAR-10 dataset using the ResNet-18 model with the non-IllD setting of the Dirichlet Distribution (a 0.6). Figure presents accuracy and loss curves of FedAvg and our FedPhoenix on the training dataset and testing dataset. From Figure 1, we can observe that although the global model trained by FedAvg can eventually achieve 100% accuracy and close to loss in the training dataset, its performance is still seriously limited in the testing dataset. We can find that the loss in the testing dataset initially decreases and then increases as training progresses. Based on the above observations, we can find that FedAvg-base FL easily overfits to the training dataset.
$
MOVE dmerston?
Cemtanae 3WS:
-20 dimenston
HOME dimenston
1-SNE dimension
(a) Original (Drrain)
(b) Masking (Dreain)
(c) Original (Drest)
(d) Masking (Drest)
Figure 2: t-SNE for representations of the model trained by FedAvg
Study on Learned Features of the Global Model. To investigate whether the global model re- lies on small number of overfitted features, we stochastically masked 10% of the parameters in FedAvg- reddAvg-triined trained global model to remove certain features. We then collected intermediate representa- tions from both the original and the masked models using the training and test datasets, categorizing them by label. Figure 2 illustrates the similarities between the representations of the original global model and the masked model on both the training and test datasets using t-SNE [36]. We observe that, in the training dataset, the intermediate representations of the original model across different
3

---

categories form distinct clusters. Nonetheless, when 10% of the parameters are masked, it becomes challenging to differentiate these clusters. In the test dataset, the representations of the original model across different categories overlap to some extent, indicating less distinct clustering. After masking 10% of the parameters, the representations of categories in the test dataset become nearly indistinguishable. Therefore, masking small number of features can cause significant seri- ous performance degradation in the global model trained via FedAvg-based FL.
To further investigate the impact of a small subset of features on models trained by FedAvg, we evaluate the importance of each feature in a specific layer on the CIFAR-10 test dataset for each class by individually masking a single feature and measuring the accuracy degradation for that class. Then, we test the accuracy on the test dataset for the model trained by FedAvg with masking the top 16 important features (accounting for 3. 3.125% of the total features) and ii) the stochastic 16 features of the last convolutional layer, respectively. As shown in Table 1, for models trained with FedAvg, masking the top 16 features results in significant accuracy degradation, while masking the stochastic 16 features has little effect on accuracy. Based on the above observations, we find that the inference the global model of FedAvg relies only few specific features rather than more generalized ones, making it vulnerable to parameter masking.
Our Idca. Motivated by the aforementioned observations and results, this paper aims to guide the model to acquire more generalized features and to make inferences using a wide range of features rather than relying on just few specific ones. To achieve this goal, we propose FedPhoenix, which randomly resets small subset of the global models parameters to generate local models for FL training. In this way, the cloud server can destroy a few features of the global model. Since most clients can learn generalized features, these features can be quickly restored. On the contrary, the specific local features unique to a few clients are hard restore. In addition, since features with large weights randomly reset, parameter resetting can guide the model to adjust its weights to learn more features for inference rather than just few.
Table I: Impact of Partial Feature Layer Output Masking on Class-wise Accuracy
Accuracy of Fed (%)
Class
Original
Topló
Random16
Class 0
64. 14.10%
38.80%
63.30%
Class
70.80%
47.30%
72.30%
Class 2
54.60%
33.40%
54.10%
Class
41.70%
22.10%
43.10%
Class
64.40%
39.40%
64.50%
To validate the effectiveness of our idea, we compare the performance of FedPhoenix with that of FedAvg. As shown in Figure I, FedPhoenix can greatly alleviate overfitting and achieve higher inference accuracy on the test dataset.
KOME smensin?
2 Uuquoou 2004
20.
20
3N51
3451
-20
DOME dimeniion
I-SNE demerssnn
dimmessoo
19N7 dimersoen
Original (Drrain)
(b) Masking (Dreain)
(c) Original (Drest)
(d) Masking (Drea))
Figure 3: t-S for representations of the model trained by FedPhoenix
Figure 3 visualizes the similarities of representations of the original global model and the masked model trained by FedPhoenix on the training dataset and test dataset using t-SNE. We observe that representations the original and the masked models across different categories form distinctly separate clusters in both the training and test datasets, indicating that the global model trained by FedPhoenix learns more generalized features and utilizes more features for inference than the model trained by FedAvg.
4
Our Phoenix Approach
4.1
Overview
Figure 4 illustrates the framework and workflow of our FedPhoenix approach, which includes central cloud server and multiple local clients. In each FL training round, the cloud server selects K

---

Cloud Server
Template Generation
Global
Model
Parameter Resetting
Model Aggregation
Reset Values
Dispatching
Uploading
Local
Local Data
Local
Data
Training Client #1
Training Client #2
Training
Client #K
Local Clients
Figure 4: Framework and workflow of FedPhoenix
activated clients to participate in local training. Assuming that there N clients, have K < N. Unlike traditional FedAvg-based methods, which directly dispatch the global model to each client, FedPhoenix first duplicates the global model into K copies and then stochastically resets a few parameters in each copy. Then, the cloud server dispatches the model copies to activated clients, where each client uses its received copy and local data conduct local training. After local training. the cloud server collects all local models and aggregates them to generate new global model. Specifically, the workflow of each FedPhoenix training round consists of five steps as follows:
Step 1: Parameter Resetting. The cloud server duplicates the global model into K independent copies and stochastically resets a subset of each copy's 's parameters. The reset proportion of each layer is dynamically adjusted based on the initial value 1io and the current training round r.
Step 2: Model Dispatching, The server sends the reset model copies to K activated clients, where each client only receives one copy.
Step 3: Local Training. Activated clients perform model training using their local data.
Step 4: Model Uploading. Each activated client uploads its trained local model to the cloud server.
Step 5: Model Aggregation. The cloud server aggregates all the uploaded local models to generate new global model, which used for the next round FL training.
4.2 Implementation
Algorithm 1 FedPhoenix Framework
Algorithm presents the implementation of FedPhoenix. As shown in Algorithm 1, Line initializes the global model was. Lines 2-12 show the FL training process, which consists of round training rounds. Line 3 randomly se- lects K clients from the client pool C, where S. is the set of selected clients. Lines 4-7 du- plicate the global model and reset the model copies, where in Line 6, the function Reset(-) resets the t model copy, and such model will be dispatched to the 1tth activated client in Sc. Lines 8- present the local training process, where in Line 9, each activated client uses their local data to train the dispatched model woooal and W.. is the trained local model. Line 11 ag- gregates all the trained local models generate the new global model w glb.
Input: i) round, # of training rounds; ii) C, the set of involved clients; iii) K, of activated clients; iv) reset proportion Output: Wglh, the trained global model.
rs. # of rounds for reset;
FedPhocni(rouund, Cattaarrooddd.,,,..e. (.r...,,o))
I: w +- Initialize global model
2: for I,,,, rownd do
S. Randomly select clients from
4:
for k= do
5:
wooccal w gib
6:
w <- Reset(w Local 0, r.))
7: end for
parallel for block */ for l: I... do
9:
10:
wi. + LocalTraining(u Local Sclk)
11: end for
12: uiit
13:
end
14: return wi.

---

4.2.1 Parameter Reset
Algorithm 2 details the model reset process. As shown in Algorithm 2, Line 2 extracts all the con- volutional layers in the target model w'. Lines 3- 13 perform the parameter reset for the target model w'. Line 5 adopts dynamic stabilization strategy to determine whether to reset the current layer according the index of the current layer i, the current round r, and the threshold rs. Line 7 calculates the number of reset kernels according
Algorithm Model Reset
Input: i) w, model weights; ii) 8, reset proportion; iii) current round; iv) rs. # of rounds for reset; Output: u², reset model weights.
Reset(ur, (w, Jo, T.)
1: w' <-
2: S t Convolutional layers in w'
3: for i = 1,2.... |S: do
4:
+ S
to the reset portion /Ag- Line randomly selects
5:
if r I iXTz then
ky kernels in 4 for reset. Line 10 resets the param-
TS;
6:
a; + number of kernels in layer
eters of the selected kernel, where w/, [k] denotes
7:
k; + ( x nt.
the corresponding parameters of the kernel &o in the w'. Here, we use normal distribution to sam- ple from the current layer's mean and variance, 10:
8:
Sp + Randomly select & kernels in 4
9:
for k E Ss do
w/, [A] Sampling(4)
and use the sampled values to reset the parame-
11: end for
ters rather than setting them zero. The sampling
12: end if
process can be defined as follows:
13: end for
14: return w'
Sampling(4) N (P4,
(1)
where and Ø; denote the mean and standard deviation of the parameters of (,, respectively. This design limits the range resets and satisfies the counded-varianee requirements that ensure the convergence of FedPhoenix.
4.2.2 Dynamic Stabilization Strategy
FedPhoenix employs a dynamic stabilization strategy to maintain stable training and convergence by adjusting the number of reset layers. Initially, more parameters reset promote the acquisition of generalized features. As training progresses, the frequency of parameter resets decreases, enabling the model to stabilize its learned features. Since each neural network layer relies the output of the previous layer, this strategy progressively reduces resets in shallower layers. As shown in the Line of Algorithm 2, assume network with [S|] convolutional layers and the 7th convolutional layers, when the current round < ixr, the cloud server will perform parameter reset for 4, where r, is [SiT
hyperparameter, However, under the dynamic stabilization strategy, as the number of model layers participating in Reset gradually decreases, FedPhoenix still maintains strong accuracy in scenarios with large number of clients.
Convergence Analysis
To demonstrate the convergence of our FedPhoenix approach (assuming that the entire equipment is involved), we present the corresponding convergence analysis. Inspired by the convergence analysis of FedAvg [37], we present the following assumptions:
Assumption 1. For E {1,2, KK, is L-smooth, where
[V/ (W)((/)]] L| w'l ww.w'.
Assumption 2. The variance of stochastic gradients is bounded:
V () VFL(W)] o Vk, w.
Assumption 3. The expected squared norm of gradients is bounded:
B V E < G² Vk,w. w.
Assumption 4.
The divergence between local and global objectives is bounded:
VF.(w)
VF(w)] 88
K

---

Based on the above assumptions and Lemmas and 2 in B.2, we have Theorem 5.1: Theorem 5.1 (Convergence Guarantee). The FedPhoenix algorithm achieves the following asymp- totic convergence rate:
lim Dg = mーt
(2)
[[F(wr)] - F'
2B 1
+
14 E
w' 12)
? ++
matching vanilla FedAvg's rate while enabling transient exploration. Please refer to Appendix B for the full proof.
6 Experiments
6.1 Experimental Settings
To validate FedPhoenix under diverse data heterogeneity scenarios, we partition each dataset via IID sampling and non-III Dirichlet Distribution [38]. By default, we assumed that only 10% of the clients participate in each FL communication round. For all experiments, optimizer with a learning rate of 0.01 and a momentum of 0.5. Each FL training round utilizes a batch size of 50 and 5 local training epochs. For data heterogeneity, we used the Dirichlet distribu- tion [39] Dir(3). where is hyperparameter to control the degree of data heterogeneity. Note that smaller /3 indicates a higher degree of data heterogeneity. We selected five baseline methods, i.c., FedAvg [1]. FedProx [40], FedGen [23]. ClusteredSampling [28], and FedMut [26]. Here, FedAvg the most classical FL method, while the other four methods are SOTA FL methods. Specifically, FedProx is a global control variable-based method, FedGen a KD-based approach, ClusteredSam- pling is device grouping-basee based method, and FedMut is mutation- based method. For FedPhoenix, we set the reset rate to = 32 for both IID and non-IID scenarios with 3 = 0.6, and set = 6 for non-IID scenarios with workstation with an Intel i9 CPU, 256GB of memory, and an NVIDIA RTX 4090 GPU.
employed the SGD
0.3. All the experimental results were obtained from an Ubuntu
Table Test Accuracy Comparison for Both Non-IID and IID Scenarios Using Three DL Models
Hetero Settings
Accuracy of Different Approaches (%)
Model
Dataset
FedAvg
FedProx
CluSample
FedPhoenkx
3=333
58.9220.22
58,8010039
64.0420.33
58.0620
00.31:0:23
S0. 50.1210.54
CIFAR-10
3 ==00 0.6
64.4410.15
65.1111000
67.78:0.09
63.5510019
65.56:10.15
82.28-10.82
HD
64.12.1.03
64.75.1.17
67.61
64.85
64.47.0.122
84.85 0.07
0.3
41,45:0.13
40.23-5041
44.45:0008
39.60-50116
40.82-00.24
56.58-1 0.20
Resnct
CIFAR-100
3 = 0.6
42.98:10.08
43.88:10.07
46.53-20.02
43.88-20.00
42.1340.12
58.86-10.17
HD
42.9710.05
42.3710.16
44.6810.25 29
42.5910.144
42.59-10.19
62.5110.01
0.3
48.7710.10
48.5210.12
50.3 0.04
48.4610.11
48.10100.06
59.041 0.03
Tiny- ImageNet
48.98-+0.1
59.74+4.09
IID
49.56=00.05
49.18:+0.05
50,56=0.03
49,46=2.04
48.26-t0.04
60.96:t0.12
8 0.3
75.92+1.116
77.33+-0.10
79.62-0.47
75.36-+0.13
76.88-40.55
81.46- 0.70
CIFAR-I0
19.17
78.23++0.09
80.45-0.199
78. 16:10.37
79 69.6640-005
83.24-1-0.5
II
80.85
10.1611.001
81.4620.01
79.98-1001
80.47:1..00
87.17-10.00
55.02:0.055
54.8910.37
16.51
$4,5310.74
38.84:10007
VGG
CIFAR 100
3
0.6
56.40-10.10
56.97
57.8% 0.34
56.89 4: 0.58
55.42-10.10
59.85 0.25
13:-0001
58.594-0.1
59.88-00001
58.53+-0.08
56.5440..00
61.00):-0.12
0.3
44.6210.06
44.2810.11
47.05=0.04
44.3820.06
51.35+0.13
56.89-10.06
Tiny-ImageNo
45.21
45.13:10.12
47.9410.066
50.89:0.02
$1.45:1.12
56.99.10.11
HD
45.04:10.01
44.85100.1
47.00
1..26
51 10.099
58. 0.02
== 0.3
30.7113.79
31,981147
36.2211.10
50,2913.44 49
$1,39:13.3
73.54:10.4
CIFAR-10
57.96
59.8910.63
62.2111.07
58.80.10.64
UD
60.91 +-0.13
61.18--0.15
64.40-0006
62.734-0.15
61.034-0.00
82.50 0.02
35.6240022
18:0.2
38,4920.14
36.15:50.31
36.054:..1
M1
CIFAR-100
3 0.6
39.32:10.05
39.6810.08
40.75=00008
39.23-10.10
38.09
53.13-1...14
HD
41.5011022
40.97.10.15
46.49.10188
40.89:10.06
39.67-10.09
$7.35.10.11
39,99110006
39.97=00077
42.42-10.05
39,9911005
40.12:10.09
03.74:10.11
ImageNea
3 0.6
41.15
41.20110.06
43.69:0004
41.4610.03
41.15
54.87-10.04
40.78-10.01
40.27
6.2 Performance Comparison
Table compares classification performance between FedPhoenix and the baseline models, evalu- ated under both IID and non-IID conditions, across different datasets and Deep Learning (DL) mod-
7

---

els. From Table 2, we observe that FedPhnoeix achieves the best performance across all cases. We observe that FedPhoenix achieves significant accuracy improvements in both IID and two non-III scenarios, demonstrating its effectiveness in hata-heterogeneous settings. We can see that, compared to cases using the VGG-16 16 model, FedPhoenix achieves greater improvement on the ResNet-18 18 and MobileNet-V V1 models. This is because our parameter reset strategy mainly operates on feature ex- tractors. FedPhoenix performs better on networks that primarily rely on convolutional layers than on dense networks like VGG-16, 16, which have more fully connected layers. It is important to note that FedPhoenix continues to outperform baseline models by a significant margin when using VGG-16.
6.3 Compatibility Analysis
Impact of Differrnt Number of Activated Clients. We conducted experiments to examine the impact of different numbers of activated clients (i.e.. K E {1. 10, 20, 50, 100}) on FedPhoenix. Figure compares the training performance between FedPhoenix and the five baselines, considering different numbers of activated clients. We observe that FedPhoenix still significantly outperforms all baselines across all cases. In addition, we observe that when fewer devices are activated, FedPhoenix exhibits significant fluctuations in accuracy during the early training stage, yet it still converges as training progresses.
Morth
(a) K
(b) K 10
(c) K 20
(d) K 50
(e) K- 100
Figure 5: Learning curves on ResNet-18 for different number of active clients with a = 0.3
Impact of the Total Number of Clients. To explore the impact of the total number of clients, we conducted experiments with different configurations of the total number of clients (i.e., |C| e (50, 100, 200, 500)). Figure presents the learning curves of FedPhoenix and five baselines with configurations of the number of clients. We observe that FedPhoenix still significantly outperforms all baselines across all cases. We can still observe that when |C| 500, FedPhoenix should spend more training rounds to achieve the highest accuracy. This is mainly because, as the number of clients increases, the amount of data available per client decreases, inherently limiting the quality of updates each client can contribute. Nonetheless, FedPhoenix can maintain comparable accuracy even with large number of clients.
Peeice
Gormuneation Aounda
Commonicuttnn Rourddy
Conmeniaulin Rounda
Conmoniaationn Rounds
(a) |C| = 50
(b) [C| = 100
(c) |C| = 200
(d) |C| = 500
Figure 6: Learning curves for different number of clients CIFAR- 10 dataset with 3 = 0.6
Compatibility with General Overfitting Mitigation Methods. To evaluate the compatibility of FedPhoenix with general overfitting mitigation methods, we conducted experiments on FedPhoenix and five baselines using two overfitting mitigation techniques in local training, all based on the CIFAR-100 dataset. First, we added Dropout layers after each of the four blocks of ResNet18 18 and before the fully connected layer, with dropout probability of p 0.5, as suggested in the original Dropout paper [41]. The results, shown in Table 3, indicate that FedPhoenix achieves the best performance in both IID and Non-IID scenarios. In both NID (with 3 = 0.3) and IID settings, FedPhoenix shows a performance discrepancy of merely 2.41% %2 in contrast to the baselines, which range from 9.43% to 15.96% This highlights its strong compatibility with Dropout and its superior adaptability to Non-IID conditions. Additionally, we evaluated the performance of ResNet-18 with weight decay, set to 0.0004. This value was chosen based on recommendations from [42], As shown
8

---

Table 3:
Test Accuracy Comparison for ResNet-18 with Dropout Layers on CIFAR-100
Hetero
FedAvg
FedProx
FedMut
CluSamp
FedGen
FedPhoenix
3-0.3
1..8
48.2411100
53,42:12.39
43,97=1.46
49.41
69.1011..35
3=0.6
04.49+00.29
52.76+0.41
53.46+00666
54.19=0.74
58.15+0.80
69.03+0008
III
58.1910.21
57.67:1..39
66.8710.13
59.93=0.18
62.51=0.09
7..51:0005
Table 4: Test Accuracy Comparison for ResNet-18 with Weight Decay of 0.0004 on CIFAR-100
Hetero
FedAvg
FedProx
FedMut
CluSamp
FedGen
FedPhoenix
3=0.3
41.73.10.41
41.23
44.59.10.25
41.79-0.09
43.10.10.09
56.73 1..45
3=0.6
44.07=+0.20
43.92+0.14
47.99:0.11
43.58=0.06
04.26=0.11
5..17=00.06
43.10:10.02
42.91110.03
47.18.00.08
43.64-0.03
43.95:10.03
6.00
in Table 4, FedPhoenix consistently outperforms others both IID and Non-IID settings, further confirming its compatibility with general overfitting mitigation methods.
6.4 Ablation Studies
Impacts of Hyper-parameters. We performed ablation studies analyze the effects of hyperpa- rameters on FedPhoenix. specifically examining different reset rounds тs and reset rates 0 on the CIFAR-10 dataset non-IID setting (B 0.3) with MobileNet-VI. Figure 7(a) presents the learn- ing curves of FedPhoenix with different values of r.s where "FedPhoenix 1000" denotes the variant of FedPhoenix with r, 1000. We can observe that small value of r, results in performance degradation of FedPhoenix. This is because a few rounds of resetting result in the model not yet learning generalized features. In addition. we can find that as r, increases, the performance of Fed- Phoenix improves. Figure 7(b) presents the learning curves of FedPhoenix with different values of We can find that a large value of results in significant fluctuations in accuracy and small value of 0 results in the accuracy degradation. However, all the variants of FedPhoenix will eventually converge and still outperform all the baselines.
Communtaiion Rounds
CommunicO Rouinds
Conmrnadiiion Riunds
(a) Reset Rounds
(b) Reset Rates 0
(c) Sampling
(d) Stabilization
Figure 7: Learning curves of different ablation studies on CIFAR-IO with 8 = 0.3
Impacts of Sampling and Stabilization Strategies. To evaluate the effectiveness of FedPhoenix's 's sampling and stabilization strategies, we conducted ablation studies on the CIFAR-I0 dataset under non-lID condition (with 3 0.3) using ResNet-18 our model. We compared our sampling strategy to the Kaiming normal distribution (a.k.a., Kaiming-Normal) Figure 7(c) demonstrates that our sampling strategy outperforms the Kaiming-Normil approach. Additionally, Figure 7(d) displays the learning curves for FedPhoenix alongside a variant that does not employ the stabilization strategy. The variant without stabilization exhibited greater fluctuations and lower accuracy than the full FedPhoenix implementation.
6.5
Computational Overhead Analysis
To provide comprehensive view of our method's efficiency, we analyzed its theoretical compu- tational overhead terms of Floating Point Operations (FL.Pss.. This analysis shows that our server-side reset mechanism incurs a modest and manageable cost.
Key Assumptions and Our Method's FLOPs. Our analysis is based on the following assump- tions: computing the layer-wise mean μ4 and variance of requires approximately 2 and 4 FLOPs per parameter, respectively. Sampling new parameters is conservatively estimated to take around 10 FLOPs per parameter. The additional overhead of our method occurs entirely on the server side. This includes one-time computation of statistics, which is estimated at about 6MM FLOPs (where M represents the total number of model parameters), and a per-client reset operation, estimated at

---

Table 5: Comparison of per-round computational overhead (AFLOPs) relative to vanilla FedAvg.
Our method involves modest server-only cost, unlike methods that require significant client-side computation or struggle to scale with the total number of clients N.
Method
Overhead Source & Location
Per-Round AFLOPs
FedPhoenix (Ours)
Server: Statisties computation & parameterr sampling for K clients.
M + 109K)
FedProx
Client: Proximal termn computation in each of the P local steps.
2s 3PKM
FedMut
Server: Global gradient aggregation & model mutation K clients.
M(1 + 2/)
ClusteredSampling
Server: Pairwise similarity computation hetween K models and all N clients.
3KNM
FedGen
Both: Multi-step generator training (Server) & knowledge distillation (Client).
High"
Notations: M. model parameters; K, the number of participating clients per round; N, the number of total clients (N K); P. the number local update steps; perturbation ratio.
"FedGen's overhead involves multiple forward/backward passes on both the client and server, often exceed- ing those of other methods by orders of magnitude.
approximately 100M FLOPs for each client, with being the perturbation ratio and K referring to the number of clients. Consequently, the total overhead per round is given by M(6 + 10/K).
Comparison with Baselines. Table 5 compares the additional computational overhead per round (AFL.OSs of method with the baseline methods. This comparison demonstrates that our server- only overhead is highly competitive, as it avoids the significant client-side costs and the poor scala- bility associated with the total number of clients (N) observed in other methods. The results show a clear efficiency advantage for our approach. For a ResNet- 18 model (M R 11M) with K = 10 and 0 0.2, our total reset FLOPs are 286M On an NVIDIA A100 GPU, this adds a negligible wall-clock time of less than 0.1 seconds per round. This is significantly more efficient than meth- ods like FedProx or FedGen, which impose heavy computational burdens on resource-constraing clients, and scales better than ClusteredSampling in scenarios with large pool of total clients N.
6.6 Limitations
While the above experiment results are promising, certain limitations point to areas for future re- search. First, our methods and experiments primarily focus vision tasks and Convolutional Neu- ral Networks (CNNs). It remains uncertain whether the FedPhoenix framework can be applied to other domains, such Natural Language Processing (NLP) tasks and Transformer architectures. We plan to adapt the FedPhoenix framework for these models and tasks in our future studies. Addi- tionally, the convergence analysis presented here is limited scenarios where all devices participate fully. The stage-wise perturbation decay mechanism indicates that the perturbation resulting from the reset operation will eventually decay to zero, even with partial participation. This could lead to convergence behavior similar to that of FedAvg; however, this theory needs comprehensive theoret- ical analysis and experimental validation.
7 Conclusion
This paper introduces FedPhoenix, a novel FL framework that incorporates heuristic parameter reset strategy to improve the generalization of the global model in non-lII scenarios. By randomly resetting certain subsets of global model parameters, FedPhoenix disrupts client-specific features and promotes collaborative restoration of more generalized features across clients. Theoretical anal- ysis shows that FedPhoenix converges effectively, and experiments conducted on three benchmark datasets demonstrate its effectiveness.
Acknowledgment
This work was supported by the Natural Science Foundation of China (62272170). the Shanghai In- ternational Joint Lab of Trustworthy Intelligent Software (22510750100), and the National Research Foundation, Singapore, and Cyber Security Agency of Singapore under its National Cybersecurity R&D Programme and CyberSG R&D Cyber Research Programme Office. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, Cyber Security Agency of Singapore as well as CyberSG R&D Programme Office, Singapore.
10

---

References
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Ar- cas. Communication--fficii learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[2] Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. Feder- ated learning for generalization, robustness, fairness: A survey and benchmark. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024.
[3] Wenke Huang, Mang Ye, Zekun Shi, and Bo Du. Generalizable heterogeneous federated cross- correlation and instance similarity learning. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 46(2):112-75) 2023.
[4] Ming Hu, Zeke Xia, Dengke Yan, Zhihao Yue, Jun Xia, Yihao Huang, Yang Liu, and Ming- song Chen. Gitfl: Uncertainty-aware real-time asynchronous federated learning using version control. In 2023 IEEE Real-Time Systems Symposium (RTSS), pages 145-157. IEEE, 2023.
[5] Lele Fu, Sheng Huang, Yanyi Lai, Tianchi Liao, Chuanfu Zhang, and Chuan Chen. Beyond federated prototype learning: Learnable semantic anchors with hyperspherical contrast for domain-skewed data. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 16648-16656, 2025.
[6] Yichen Li, Qunwei Li, Haozhao Wang, Ruixuan Li, Wenliang Zhong, and Guannan Zhang Towards efficient replay in federated incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12820-12829, 2024.
[7] Tianchi Liao, Lele Fu, Jialong Chen, Zhen Wang, Zibin Zheng, and Chuan Chen. A swiss army knife for heterogeneous federated learning: Flexible coupling via trace norm. Advances in Neural Information Processing Systems, 37:139886-1399 2024.
[8] Chentao Jia, Ming Hu, Zekai Chen, Yanxin Yang, Xiaofei Xie, Yang Liu, and Mingsong Chen. Adaptivefl: Adaptive heterogeneous federated learning for resource-constrained aiot systems. In Proceedings of the 61 ACM/IEEE Design Automation Conference, pages 1-6, 2024.
[9] Zeke Xia, Ming Hu, Dengke Yan, Xiaofei Xie, Tianlin Li, Anran Li, Junlong Zhou, and Ming- song Chen. Cabafl: Asynchronous federated learning via hierarchical cache and feature bal- ance. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 43 (11):4057-4068, 2024.
[10] Zekai Chen, Chentao Jia, Ming Hu, Xiaofei Xie, Anran Li, and Mingsong Chen. Flexfl: Het- erogeneous federated learning apoz-guides flexible pruning in uncertain scenarios. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 43(11):4069- 4080, 2024.
[11] Cheng Dai, Shuai Wei, Shengxin Dai, Sahil Garg, Georges Kaddoum, and M Shamim Hos- sain. Federated self-supervised learning based on prototypes clustering contrastive learning for internet-of-vehiiles applications. IEEE Internet of Things Journal, 2024.
[12) Dengke Yan, Yanxin Yang, Ming Hu, Xin Fu, and Mingsong Chen. Mmdfl: Multi-model- based decentralized federated learning for essurcc-constraiiee aiot systems. In 2025 62nd ACM/IEEE Design Automation Conference (DAC), pages 1-7. IEEE, 2025.
[13] Zhufeng Lu, Chentao Jia, Ming Hu, Xiaofei Xie, and Mingsong Chen. Gradients as an action: Towards Communication-effiicee federated recommender systems via adaptive action sharing. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 1999-2009, 2025.
[14] Yichen Li, Yijing Shan, Yi Liu, Haozhao Wang, Wei Wang, Yi Wang, and Ruixuan Li. Per- sonalized federated recommendation for cold-start users via adaptive knowledge fusion. In Proceedings of the ACM on Web Conference 2025, pages 2700-2709, 2025.
11

---

[15] Rodolfo Stoffel Antunes, Cristiano André da Costa, Amn Küderle, Imrana Abdullahi Yari, and Bjöm Eskofier. Federated learning for healthcare: Systematic review and architecture proposal. ACM Transactions on Intelligent Systems and Technology (TIST), 13(4):1-23, 2022.
[16] Zelei Liu, Yuanyuan Chen, Yansong Zhao, Han Yu, Yang Liu, Renyi Bao, Jinpeng Jiang, Za- iqing Nic, Qian Xu, and Qiang Yang. Contribution-awant federated learning for smart health- care. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12396-1244 12404, 2022.
[17] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of International Conference on Machine Learning (ICML), pages 5132-5143.
PMLR, 2020.
[18] Zhuang Qi, Lei Meng. Zitan Chen, Han Hu, Hui Lin, and Xiangxu Meng. Cross-silo proto- typical calibration for federated learning with non-iid data. In Proceedings of the 31st ACM International Conference Multimedia, pages 3099-3 3107,317700000000000000011 2023.
[19] Zhuang Qi. Sijin Zhou, Lei Meng, Han Hu, Han Yu, and Xiangxu Meng. Federated decon- founding and debiasing learning for out-of-distribution generalization. In The 34th Interna- tional Joint Conference on Artificial Intelligence, (5C) pages 2025.
[20] Yanbiao Ma, Wei Dai, Wenke Huang, and Jiayi Chen. Geometric wwwwlldge-guidenn localized global distribution alignment for federated learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
[21] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[22] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems, 33: 2351-2363, 2020.
[23] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heteroge- neous federated learning. In Proceedings of International Conference on Machine Learning (ICML), pages 12878-12889. PMLR, 2021.
[24] Haozhao Wang. Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain-aware federated knowledge distillation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages
2023.
[25] Siran Zhao, Tianchi Liao, Lele Fu, Chuan Chen, Jing Bian, and Zibin Zheng. Data-free knowl- edge distillation via generator- free data generation for non-iid federated learning. Neural Net- works, 179: 179:106627, 2024.
[26] Ming Hu, Yue Cao, Anran Li, Zhiming Li, Chengwei Liu, Tianlin Li, Mingsong Chen, and Yang Liu. Fedmut: Generalized federated learning via stochastic mutation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22528-12377, 2024.
[27] Jiawen Weng, Zeke Xia, Ran Li, Ming Hu, and Mingsong Chen. Fedqp: Towards ac- curate federated learning using quadratic programming guided mutation. arXiv preprint arXiv:2411.15847, 2024.
[28] Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. Clustered sampling: Low- variance and improved representativity for clients selection in federated learning. In Interna- tional Conference on Machine Learning, pages 3407-3416. PMLR, 2021.
[29] Anran Li, Guangjing Wang, Ming Hu, Jianfei Sun, Lan Zhang, Luu Anh Tuan, and Han Yu. Joint client-and-sampll selection for federated learning via bi-level optimization. IEEE Trans- actions on Mobile Computing, 2024.
12

---

[30] Cheng Chen, Ziyi Chen, Yi Zhou, and Bhavya Kailkhura. Fedcluster: Boosting the conver- gence of federated learning via cluster-cycling, In 2020 IEEE International Conference Big Data (Big Data), pages 5017-5026. IEFE, 2020.
[31] Zhuang Qi, Yuqing Wang. Zitan Chen, Ran Wang, Xiangxu Meng, and Lei Meng. Clustering- based curriculum construction for sample-balancee federated learning. In CAAI International Conference Artificial Intelligence, pages 155-166. Springer, 2022.
[32] Ming Hu, Zhihao Yue, Xiaofei Xie, Cheng Chen, Yihao Huang, Xian Wei, Xiang Lian, Yang Liu, and Mingsong Chen. Is aggregation the only choice? federated learning via layer-wise model recombination. In Proceedings of ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 1096-1107, 2024.
[33] Ming Hu, Peiheng Zhou, Zhihao Yue, Zhiwei Ling, Yihao Huang, Anran Li, Yang Liu, Xiang Lian, and Mingsong Chen. Fedcross: Towards accurate federated learning via multi-model arossaaggreg In Proceedings of IEEE International Conference on Data Engineering (ICDE), pages 2137-2150. IEEE, 2024.
[34] Zeke Xia, Ming Hu, Dengke Yan, Ruixuan Liu, Anran Li, Xiaofei Xie, and Mingsong Chen. Multisfl: Towards accurate split federated learning via multi-model aggregation and knowledge replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2025.
[35] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy-preserving machine learning. In proceedings of ACM SIGSAC Conference on Com- puter and Communications Security, pages 1175-1191,2011
[36] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11), 2008
[37] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the conver- gence of fedavg non-iid data. arXiv preprint 1rXiv:1907.02189, 2019.
[38] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heteroge- neous federated learning, 2021. URL https:
org/ 10056.
[39] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification, 2019. URL https: /tatpp:///rxii.... org.ooog//a.. /ba//
1909. 003335.
[40] Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In European Conference on Computer Vision, pages 654 672. Springer, 2022.
[41] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Ma- chine Learning Research, 15(56):1929-199) 1958, 2014. URL http::/jmmlt wrt.a//rrrr/pppprr//ss..s srivastava14a. html.
[42] Taehwan Lee and Sung Whan Yoon. Rethinking the flat minima searching in federated learning. In Forty-first International Conference on Machine Learning, 2024. URL https: / //openreview net/ http:/formmiddddfooooa
[43] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-4,, 1997.
[44] Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent, 2016. URI. httts:://arxis http://awwwwv...a/////eeeees wrw./bs//50//// 01240.
[45] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the conver- gence of fedavg non-iid data, 2020. URL ttttps://ariic .rg/ wwwp///////////////e../
13

---

A Additional Experimental Results
A.1 Learning Curves of Different FL Methods
10:00
1993
1200
431
153
TE00
Communication Rounds
Communication Rounds
Communication Rounds
(a) MobileNet-VI (8 0.3)
(b) MobileNet-Vi (3 006)
(c) MobileNet-Vi (IID)
Acauncyy
Acsuraty
Foskig
Fadiog
Fadtin
Parton
FisPics
Fadhoo
FalPharex
Communication Rounds
Communicattion Rounds
Communicaiion Rounds
(d) R8 0.3)
(c) RessNet-18 (B 0.6)
(f) ResNet- 18 (IID)
Acoumto
Accuraoy(S)
Clntracianpiigg
Qustrnc Sanaanngg
Fiskig
Tackin
Futtla
4P0
1080
400
500
1966
1000
Communication Rounds
Communication Rounds
Communication Rounds
(g) VGG- (3 0.3)
(h) 16 0.6)
(i) VGG-16 (IID)
Figure 8: Learning curves comparison CIFAR-10 10 with three network architectures under different data distributions. From top to bottom: MobileNer-Vi ResNet- 18, and VGG-16. Columns show different Dirichlet concentration parameters (3 0.3, 0.6) and IID setting
A.2 Supplementary Tables of Test Accuracy Comparison (Extended from Section 6.3)
We evaluated the performance of FedPhoenix and baseline methods using a ResNet-18 architec- ture with consistent regularization measures (e.g.. Dropout and Weight Decay) on both CIFAR-10 and Tiny-Image ImageNet datasets. Experimental results demonstrate that FedPhoenix consistently out- performs all baselines, achieving state-of-the-- accuracy under identical experimental conditions.
Table 6: Comparison for ResNet- 18 with Dropout layers on CIFAR-10
10 and
Dataset
Hetero
FedAvg
FedProx
FedMut
CluSamp
FedGen
FedPhoenix
3=0.3
19.89++14.77
61.07=3.76
66.60+0.27
18.95=10.05
79.51+44.4
8..54+0.11
CI
3=0.6
75.4213.51
1..05=1.61
80.23:11338
70.7111.84
85.0611235
87.56 0.27
76.7210.47
18.9210.1
82.2710.20
75.7410.32
89.6910.05
3=0.3
49.8110.31
50.55=0.10
52.04.10.07
50.7111007
50.35.10009
57.68
Tiny-ImageNet
3=0.6
51.03100.06
51.88-0.11
52.67-00.07
52.05::0.04
52.25:10.04
59.54-10.04
11D
51,78-0.03
52.77=0.03
53.62+0.02
52.2640.03
52.12440133
61.01=0.022

---

Table 7: Comparison for ResNet-18 with Weight Decay 0.0004
CIFAR-I0 and Tiny- ImageNet
Dataset
Hetero
FedAvg
FedPrrx
FedMut
CluSamp
FedGen
FedPhoonix
5-0.3
60.2516.12
60.5110:29
64.1310.40
58.5812.65
60.08
79.20.12.51
CIFAR-10
3=0.6
66.31
64.42-40.26
70.87 +0.05
63.91
66.54
81.70+F.111
11D
65.28:00.03
64.75:0.004
67.82=0.04
65.98:0.02
67.3840.00
85.24:20.01
0=0.3
51.25++0.10 10
51.01=0.088
52.57=0.14
50.62+00.13
51.30=00.09
58.34:+0.10
Iing-emageNet
3=0.6
51.82.10.24
11.91
52.92.10.18
52.0510.115
52.89100.08
59.30-10.10
II
51.142:0.05
31.54:20:04
31,9320.06
52.26=0.04
52.56:20:01
61.291:0.02
A.3 Comparison of Generalization
According to the observations in [43, 44], the more generalized model is typically located in flat areas rather than sharp areas. To evaluate the generalization of FedPhoenix, as shown in Figure we visualized the loss landscapes of the models trained by FedAvg and FedPhoenix on CIFAR-10 10 under IID and non-IID (3 0.3) settings. We can observe that the global model trained by FedAvg is located in sharp area. In contrast, that trained by FedPhoenix converges to significantly flatter area across both settings, which demonstrates the good generalization ability of FedPhoenix.
(a) FedAvg (IID)
(b) FedPhoenix (IID)
(c) FedAvg (0.3) (d) FedPhoenix (0.3)
Figure 9: Comparison FedAvg and FedPhoenix loss landscapes
A.4 Performance under Extreme Data Heterogeneity
To thoroughly test the limits FedPhoenix, we conducted experiments under two challenging sce- narios: extreme skew in label distribution and extreme skew in feature distribution.
A.4.1 Performance under Extreme Label Skew
We evaluated our method against baselines on CIFAR-10 with highly skewed label distributions, simulated by Dirichlet distribution with very small concentration parameter (8 0.01 and / 0.1). As shown in Table 8, FedPhoenix demonstrates significant performance gains across various model architectures (ResNet-18. VGG16, and MobileNet-VI), especially the most extreme non- IID setting (3 0.01). This highlights its robustness and superior ability to mitigate client drift when local data distributions are severely biased.
Table 8: Accuracy (%) on CIFAR-10 under extreme non-llI (label skew) settings. FedPhoenix consistently outperforms all baselines, with particularly strong gains when 3 = 0.01.
Model
Setting (3)
FedAvg
FedProx
FedMut
CluSamp
FedGen
FedPhocuix (Ours)
ResNet-18
28.20
28.53
0.26
0.04
35.80
0.1
45,48 +: 0.03
45.41 + 0.16
47.55 + 0.14
45.56 +: 0.36
45.29
7.33
55.80 = 0.11
0.01
32.24 0.16
31.79 + 0.37
32.09 + 0.16
30.74 0.57
32,03
1.89
33.10 0.41
VGG16
0.1
47.75
48.26 0.18
49.59
47.72
0.29
52.86
0.14
MobileNet-V
0.01
22.24 1: 0.16
21.79 + 0.37
22.09 1 0.16
20.34 4: 0.57
21.01 1.89
27.56 = 0.57
0.1
47.75 1: 0.11
48.26 1 0.18
49.59 1 0.07
47.72 1: 0.05
37.93 0.29
50.36
0.11
A.4.2 Performance on Extreme Feature Skew (Office-31 Dataset)
To simulate scenario where clients' feature spaces are disjoint, we used the Office-31 dataset, benchmark for domain adaptation. We assigned each of the three domains (i.e., Amazon, DSLR, Webcam) to a separate client. This setup mimics federated environment with extreme feature skew. The results in Table 9 show that FedPhoenix achieves substantial accuracy improvement over all baselines. This empirically validates that our method effectively facilitates collaboration and knowledge transfer even when clients' data come from entirely different distributions.
15

---

Table Accuracy (%) on the Office-31 31 dataset, partitioned by domain simulate extreme feature skew. FedPhoenix significantly outperforms other methods.
Dataset
FedAvg
FedProx
FedMut
CluSamp
FedGen
FedPhoenix (Ours)
Office-31 47.84 + 1.78 48.12 + 1.68 43.99
1.59
46.42 + 1.83
49.13 2.26
54.17 0.96
A.5
Ablation Studies on the Reset Operation
We conducted a series of ablation studies to justify the core design choices of our reset operations.
A.5.1 Target Layers: Convolutional vs. Fully-Connected
To validate rationale for exclusively targeting Convolutional (Conv) layers, compared it with applying the Reset Operation to Fully-Connected (FC) layers only or to both. The results in Table 10 confirm our hypothesis. Resetting only Conv layers yields the best or highly competitive performance, especially in the non-IID setting. Conversely, resetting FC layers alone detrimental, causing significant drop in accuracy. This supports our design choice enhance the shared feature extractor (Conv layers) while preserving the stability the classifier (FC layers).
Table 10: Ablation study on the target layers for the Reset Operation (ResNet and VGG on CIFAR- 10). Results show that resetting only convolutional layers is the most effective strategy.
Model
Setting
Both Conv & FC
Conv Layers Only (Ours) FC Layers Only
Non-IID (8 = 0.3)
78.56 + 1.08
80.12 + 0.54
57.17 + 0.71 66.55 + 0.03
ResNet
IID
85.31 + 0.01
85.06 0.03
Non-IID (8 0.3)
82.21 + 0.28
81.46 + 0.70
76.81 + 0.27 78.92 + 0.03
VGG
IID
87.17 + 0.00
87.17 + 0.00
A.5.2 Reset Granularity: Full vs. Partial Kernel Reset
We investigated whether resetting an entire kernel is superior to resetting only fraction of its most significant parameters (by magnitude). We compared our full-kernel reset with partially resetting the top 1/3 or top 2/3 parameters within each selected kernel. As shown in Table 11, the original FedPhoenix approach of resetting the entire kernel as single functional unit consistently outper- forms partial resets. We hypothesize that full reset more effectively promotes feature exploration and diversity.
Table 11: Ablation study on reset granularity (Non-IID CIFAR-10, ResNet-18). FedPhoenix's 's full- kernel reset is compared against resetting only the top-magnitude parameters within a kernel.
Setting (3)
Top-1/3 Reset
Top-2/3 Reset
Top-1/3 (Fixed Ø)
Top-2/3 (Fixed 0) FedPhoenix (Full Kernel)
0.3
77.20 + 1.87
78.47 = 1.32
76.86 + 0.98
79.55 + 0.95
80.12 = 0.54
0.6
80.68 + 0.17
80.90 + 1.06
80.35 + 0.05
81.76 + 0.78
82.28 + 0.82
IID
83.01 + 0.01
84.78 = 0.02
84.26 + 0.01
84.63 + 0.01
84.85 + 0.07
A.5.3 Sampling Distributions for Kernel Initialization
To test the robustness of our method to the choice of sampling distribution, we conducted an exten- sive study comparing our proposed adaptive Normal distribution (Adp_M) __pppaloml with wide range of other common initialization methods. All distributions were adapted using the target layer's s statistics. The results Table show that FedPhoenix performs consistently well across tested distributions. This demonstrates that the core strength of our method lies in the adaptive reset strat- egy itselfre-son sarifrr--aapiing based on the layer's current statistical propertiesrather than dependency on specific distribution.
16

---

Table 12: Ablation study on sampling distributions for the Reset Operation (ResNet-18 18, CIFAR-1O), 10).
Performance is robust across a wide variety of distributions.
Nan-ID
Noemal
Kaining, Unifeem
Normal
Xarker
Naviar
Laphaes
80.12 2 0.54
50.0 =: 0.41
79.34 064
77.58 = 234
79.42 = 145
78.87 2:00
8000 = 0.15
78.45 0.34
0.6
52.28
8186 0.24
79.97 2-.77
80.34 =041
$1.20
30.0 2.22
$1.00 0.00
81.8 0.11
51.74 0.7
0.0
6451
0.01
B Convergence Analysis
B.1 Notations and Perturbation Mechanism
Let index the local SGD iteration, and let o2 denote the intermediate model after one SGD step.
Let T = n x E represent the total number of iterations. For layer 4 with kernels, define:
S represents the total number of feature layers.
Perturbation ratio 0 E (0, 1), resetting [On:] kernels per round.
#4 J. of Var(W,,) ) for layer parameters, where and of are the statistical mean and variance computed from the current parameters of layer 4.
Perturbation operator A ~ NWu,, of) for resampled kernels, indicating that new kernel parameters are sampled from normal distribution with mean μ4; and variance of.
The laser-wise perturbation variance is derived as follows. For each kernel j (where = 1, ...... [@n:]) that reset in layer 4, the new parameter value w sampled from N ((1; af), while the original parameter value is wj. The perturbation for single kernel is weew Wj, and the expected squared perturbation is:
E new - w/ (uueer - 14 + 4s - -w
[I [u - + +1(u 11,) + 210 - p4)( wi)l
[w 14) B[[ww -1,)
(4)
of + B[(w;; [4:]]
(5)
Here, E[(w; - [+] represents the variance of the original parameter wy around the layer mean us. Since 0 is the statistical variance computed from all parameters in layer 4, we approximate B[(w, (4)2] of for each parameter Wy in the layer. This approximation reasonable due to
the large number of parameters in typical neural network layers, where the individual parameter variance is close the layer-wise statistical variance. Thus, we have:
Bl wy of of 207
(6)
For the entire layer 4, with [On]] kernels being reset, the expected perturbation variance is:
Ong
E E|
11
E
(w)
new
mrr)²
[0ms]
20 210m:107 s 20n,o
(7)
B.2 Key Lemmas
Lemma 1 (Client Drift Bound), Following the proof structure of Lemma 3 in [45], bound the expected client drift as:
K
E
w1ll
-
40 noo
(8)
17

---

represent the total perturbation applied to client k's model across all layers due to kernel resetting at the start of communication round. The difference between the local model w of client & and the global model wy can be decomposed into contributions from the initial perturbation and the local SGD updates. Specifically, starting from the perturbed initialization, we have:
Proof. Let (
- ad A,
E-1
|uo wr|12 - 11) 12 +
Σ c=0
thtt V FF St:ee
Sete)
Taking expectations on both sides:
For the perturbation term, using the layer-wise perturbation variance derived earlier (Equa- tion 7), we have:
E 1| B
(k)
E
E
(k)
o0 ? 20 noo?
For the local SGD update term, under Assumption (bounded gradient norm), we follow the standard analysis in [45]:
E 1
Σ
Moto Fu Setee
S 1E
c
Combining these bounds into Equation 9, obtain:
led
w.ll
2 20
noo +2 2
40 leel
noo + EE
(10)
Summing over clients & = 1,2, ...,. K, and noting that the bound holds uniformly for each client, arrive at the desired result:
K
+ 40 s
E
I well En -
noa
(1I)
Lemma 2 (One-Stey Step Descent), Let L/p. For learning rate π) a(rte) satisfying the conditions in [45], we have:
lxd
E3 Elluu411 - =1 (1-11/)))llu - w²1l + + B + 40
> bonn
(12)
where B = Em- EN 6LT + 8(E - 1
Proof. The proof follows the structure of Lemma 1 in [45], which analyzes the one-step descent of SGD in the federated setting under Assumptions (L-smoothnesss) and 4 (strong convexity). The key difference FedPhoenix is the additional perturbation introduced by kernel resetting, which affects the client drift term. From Lemma I in the standard FedAvg analysis, the one-step descent bound includes terms related to variance, data heterogeneity, and client drift. Incorporating the modified led 1=1 noo? is
client drift bound from our Lemma the additional perturbation variance term 40 added to the original bound Thus, we obtain:
Ele
Elluz+11 - = < (1 mo)BB/.. w01l 112 + n/²
B
+ 40
noo?
(13)
where Σi= 12 7LT + 1)2 captures the standard variance and divergence terms from FedAvg.
D
18

---

Theorem B.1 (Global Convergence). With y > max E) and a decaying perturbation ratio (r)), the expected optimality gap of the global model after T iterations satisfies:
[(F(w)) F
R ? +
2 + 4Ds)
+ 2
177 17 22
(14)
where Dg = A |
nooe represents the perturbation variance term.
Proof. The proof follows the standard convergence analysis of FedAvg as in [45], leveraging the one-step descent bound from Lemma 2. Starting from the recursive inequality:
Isd
E1|1211 -=" S Bllr) w + rz² B + 40
Σ noo
(15)
we define A = Bllle - wº and T) (( By induction and the choice of y > max E}. which ensures 7/t 1 and 1)t s 21 we can telescope the inequality over T' iterations. Noting that 40 E noo 4Dg, the additional perturbation term is incorporated into the final bound. Using the L-smoothness of F(-), the optimality gap is bounded as:
L E[F(wT)] - F < A
K 7+T
2(B + 4Do)
17 El - -w-13)
(16)
where K = L/p.
B.3
Stage-wise Perturbation Decay
To mitigate the impact of perturbation on convergence in later stages, we adopt a stage-wise decay strategy for the perturbation ratio over communication rounds. For a predefined number of commu- nication rounds r, during which Reset perturbations are applied, the perturbation ratio ((r) at round is defined as:
+
[ Stl
((r) = 6u
(17)
where 6 is the initial perturbation ratio, and |St] represents the total number of convolutional layers
the model. This ensures that ((r) within r, rounds of applying perturbations. Substituting
the decaying (r)) into Theorem B.1, the perturbation variance term Dg = 0(r) Σ |arl ishes over time:
i=l noo dimin-
lim Do O, 2-
(18)
- F < K 12
MY E
B[F((T)]
w
(19)
T
This recovers the original FedAvg convergence rate with O(1/T) dependency as the perturbation effect vanishes.
C Discussions
C.1 Privacy Preserving
While the primary objective of FedPhoenix is to enhance model generalization and robustness, its unique parameter reset mechanism also offers notable privacy considerations, even though does not provide formal guarantees like (c, D)fDDifffeential Privacy. The core mechanism, which resets fraction (0) of convolution kernels based on the layer's statisties, introduces stochasticity into global model. This inherent randomness increases the effort adversary must expend to success- fully carry out attacks such as model inversion, as the model parameters are less deterministic across communication rounds.
19

---

Crucially. FedPhoenix is fully compatible with standard ehbancy-nn technologies like secure aggregation. Since the server-side reset operation based solely on the aggregated global model, secure aggregation protocols can be seamlessly applied to protect individual client updates from being inspected by the server.
Furthermore, we can significantly strengthen privacy protection, particularly against a malicious server, by shifting the reset operation to the client side. In this configuration, each activated client receives the same global model and then independently performs the parameter reset locally. This creates a unique training starting point for each client. Consequently, even malicious server can- not know the exact initial model from which a client's update was generated, which substantially complicates server-side reverse engineering or inference attacks targeting a specific client's data. As our computational analysis demonstrates, the overhead of the reset operation is minimal, making it entirely feasible for individual clients to bear this cost.
For future work, we plan to quantitatively evaluate these inherent privacy benefits. To achieve for- mal privacy guarantees, FedPhoenix can be further integrated with techniques such as Differential Privacy, for instance, by adding calibrated noise during the parameter sampling process. This would create comprehensive framework that balances performance, robustness, and provable privacy for secure deployment in real-world federated learning scenarios.
C.2 Limitations
Although the FedPhoenix method effectively mitigates overfitting issues introduced by distributed data, this work still has several limitations that warrant further exploration future research. First, the proposed method and experimental design primarily focus vision tasks and convolutional neural networks (CNNs), which are commonly studied in federated learning. Consequently, its applicability Transformer architectures and natural language processing (NLP) tasks remains un- verified. In subsequent work, we plan to extend the FedPhoenix framework to accommodate these emerging models and tasks, thereby improving its generalization capability.
Additionally, in the Convergence Analysis section, we only discuss the scenario of full device partic- ipation. However, considering the Stage-wise Perturbation Decay mechanism, even in partial device participation scenarios, the perturbation introduced by the reset operation will gradually decay to This suggests that the convergence behavior of FedPhoenix may be similar that of FedAvg, though this hypothesis requires more detailed theoretical analysis and experimental validation to confirm.
20

---

NeurIPS Paper Checklist
The checklist designed to encourage best practices for responsible machine learning research, ad- dressing issues of reproducibility, transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (12 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- is perfectly acceptable answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way, we acknowledge that the true amswer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
ation. While "[Yes] generally preferable to "[No]
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the papers contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification:
The paper discusses the limitations in Section ??
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides the proofs of theoretical results in Section B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper provides all the necessary information to reproduce the main ex- perimental results.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-soures dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors welcome deseribe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper provides code in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
gewdde//d//ggg /CodeSubm is asssonoooicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
23

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper provides the experimental settings in Section 6.1
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports standard deviation measure of variability for the exper- imental results that support the main claims.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
It OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides the information about compute resources in Section 6.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
24

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics bttps://neure:
Answer: [Yes]
Justification: Ethics.
The research conducted in the paper conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
25

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper uses popular datasets and models. We cites the original paper.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re- e-packagedd both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer:
[NA]
Justification:
No new assets are released in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
26

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standare components.
Please refer to our L.LM policy ltttps:/neuurps ////rnuurs.../// wwt.e/cccececce///e/aaaa for what should or should be described.
27


Reviews:
[{"review_id": "rlGRr6ZNrM", "reviewer": "Reviewer_WVfY", "summary": "This paper introduces FedPhoenix, an FL framework designed to enhance model generalization and inference performance in non-IID (non-independent and identically distributed) data scenarios. The core concept of FedPhoenix is a server-side dynamic parameter reset mechanism. This mechanism stochastically resets partial parameters of the global model before dispatching them to clients for local training, aiming to disrupt client-specific overfitting features and encourage the learning of more generalized features across the federated network. The authors contribute a novel FL framework with an adaptive parameter reset strategy, a dynamic stabilization strategy, theoretical analysis for convergence, and comprehensive empirical evaluations demonstrating significant accuracy improvements over state-of-the-art FL methods on various datasets and model architectures.", "questions": "**-** Could you elaborate on the practical implications and potential adaptations of FedPhoenix for Transformer architectures and NLP tasks?\n\nActionable Guidance: Please discuss how the parameter reset mechanism would apply to specific components of Transformers. A conceptual outline of how the destruction and restoration of features would manifest in this domain, and any anticipated challenges or necessary modifications, would be highly valuable.\n\n\n**-** Please provide a more detailed analysis of the server-side computational overhead introduced by the parameter reset mechanism.\nActionable Guidance: Quantify the computational cost (FLOPs or actual wall-clock time) of the Reset function and the overall server-side processing per round. Present this data as a function of model size (number of parameters) and the number of activated clients K to better assess scalability.\n\n\n**-** What is the rationale behind exclusively applying the parameter reset to convolutional layers and not to fully connected layers?\nActionable Guidance: Please provide a justification for this design choice. \n\n\n**-** Could you provide a more precise characterization of the privacy-preserving aspect of FedPhoenix beyond increasing uncertainty and diversity?\nActionable Guidance: Clarify whether the current mechanism offers any formal privacy guarantees against specific attack models like membership inference or model inversion.", "limitations": "In Appendix C.2, the authors acknowledge several limitations. I would argue that some of them should move to the main text. \n\n**-** The ablation studies in Figure 7 show that the performance of FedPhoenix is sensitive to the reset rounds and reset rates hyperparameters. While the paper states that all variants eventually converge, achieving optimal performance appears to depend on careful tuning. In real-world FL settings, where validation data might be limited or difficult to obtain, the sensitivity to these parameters could hint at a practical challenge for deployment.\n\n**-** The core concept relies on \"destroying\" features to encourage generalization. While effective for CNNs, the interpretation, and effectiveness of this \"destruction\" might vary significantly for different types of neural network architectures (RNN for sequential data, GNN for graph data) because of different structural implications.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n**+** The central idea of dynamic parameter reset on the server side is a genuinely innovative approach to combat overfitting in non-IID federated learning.  \n\n**+**  The paper clearly articulates the problem of models overfitting to local client data and provides compelling empirical evidence to demonstrate that existing FedAvg models indeed rely on a narrow set of specific features. \n\n**+** This design avoids the common pitfalls of client-side modifications, additional communication overhead, or the need for public proxy datasets, which often limit the practicality and deployability of other FL debiasing or generalization methods.\n\nAreas for improvement\n\n**-**  Given the increasing prominence of Transformers in FL, empirical validation is exclusively confined to vision tasks and CNNs; this significantly limits the scope of generalization implied.\n\n**-** The authors state that the server-side computational overhead for generating multiple model copies with reset parameters is comparable to FedAvg. However, the detailed quantification of this overhead is not fully convincing. For very large models or a significantly higher number of active clients (K), this overhead could become a practical bottleneck. \n\n**-** Fully connected layers are often critical for final classification and can also be prone to overfitting. The rationale for this specific focus, and whether extending the parameter reset mechanism to these layers would offer further benefits or introduce new challenges, is not thoroughly explored.\n\n**-** While introducing noise can have privacy benefits, this is a weaker claim than formal privacy guarantees (e.g., differential privacy). The paper itself suggests integrating differential privacy in future work. Clarifying the specific type of privacy enhancement offered by the current mechanism and distinguishing it from more formal privacy guarantees would provide a more accurate picture.", "was_revised": true, "final_justification": "I keep my initial score, which is already high enough. I believe the camera-ready should have sufficient time to address the issues.", "rebuttal": "### Answer to Q1/W1 (Transformer)\nThis paper primarily focuses on computer vision-based tasks. As mentioned in the Limitation section of the paper, since the different structure of Transformer and CNN, the current version of FedPhoenix cannot adapt to Transformer-based models.\n\nHowever, we sincerely thank the reviewer for this forward-thinking question. While much of the current research in Federated Learning has concentrated on CNNs for vision tasks—largely due to the proliferation of resource-constrained AIoT devices—we strongly agree that extending these privacy-preserving and collaborative learning paradigms to Large Language Models is a highly valuable research direction and a key focus for our future work.\n\n**Conceptual Feasibility: Applying FedPhoenix to Mitigate Transformer Drift**\n\nThe core philosophy of FedPhoenix is to **prevent the model from overfitting to local data by periodically resetting a portion of its learned features. These reset units, possessing high plasticity, are then compelled to rediscover more generalizable patterns.** Furthermore, the FFNs in Transformers utilize fully-connected layers. As shown in our experimental results in AQ3, when a model has a robust feature extraction layer, resetting the fully-connected structures can also lead to performance improvements.\n\nRather than a full parameter reset, which could excessively disrupt the highly structured knowledge within a Transformer, we can adapt the \"reset\" operation into a more refined **blending mechanism**. For a selected parameter `W`, the update formula would be:\n\n```\nW_new = (1 - α) * W_old + α * W_sampled\n```\n\nHere, `W_sampled` is a new value sampled from a normal distribution `N(μ, σ)` based on layer-wise statistics, and `α` is a blending factor (analogous to the original parameter `ε`, but controlling the intensity of the blend rather than the proportion of reset units).\n\nThis approach does not cause complete \"amnesia\" but rather **\"nudges\" the parameters away from a highly specialized state**. It helps to loosen the tight coupling between specific attention patterns or word embeddings and the data distribution of a single client (e.g., a \"developer\" client might associate \"Python\" with the programming language, whereas a \"zoologist\" client might associate it with the reptile), encouraging the model to build robust representations that span across client domains.\n\n**Operational Guidelines: Target Modules and Reset Granularity**:\n\n- **Self-Attention Mechanism (`W_q`, `W_k`, `W_v`, `W_o` matrices)**:\n  - **Target**: The query, key, value, and output projection matrices are prime candidates, as they directly govern the learned relationships between tokens.\n  - **Granularity**: We would apply the blending operation to entire rows or columns of these matrices. Conceptually, this is equivalent to perturbing the \"functional role\" of specific feature dimensions within the attention mechanism.\n- **Feed-Forward Networks (FFN)**:\n  - **Target**: The two linear transformation layers within each Transformer block.\n  - **Granularity**: Resetting the incoming weights for individual output neurons.\n\n**Anticipated Challenges and a Dynamic Attenuation Strategy**\n\n- **Hyperparameter Sensitivity**: The blending factor `α` and the proportion of parameters to be perturbed are critical hyperparameters that will require careful tuning to balance knowledge retention and exploration.\n- The *dynamic stabilization strategy* would be adapted as follows: **apply stronger/more frequent perturbations to the earlier Transformer blocks during the initial communication rounds.** As training progresses, the focus of the perturbation would gradually shift to the later, more abstract blocks. This approach maintains the original design philosophy of first solidifying foundational features while preserving the plasticity of higher-level representations.\n\nWe thank the reviewer again for this suggestion and will incorporate this discussion into the \"Future Work\" section of our paper to broaden its scope.\n\n### Answer to Q2/W2 (Analysis of Computational Overhead):\nSorry for the confusion. We conducted a quantitative analysis of the FLOPs (Floating-Point Operations) for each round of the Reset operation. \n\n#### Notations \n-\t$M$ denotes the total number of convolutional layer parameters;\n-\t$K$ denotes the number of activated clients, $|S_l|$ denotes the total number of convolutional layers;\n-\t$\\theta \\in (0,1)$ indicates the perturbation ratio, where $\\lfloor \\theta n_i \\rfloor$ kernels are reset per layer (with $n_i$ kernels in layer $l_i$)\n-\tFor each layer $l_i$, $\\mu_i = \\mathbb{E}[W_{l_i}]$ and $\\sigma_i^2 = \\text{Var}(W_{l_i})$ are computed from the current layer parameters;\n-\tReset kernels are sampled from $\\Delta_{l_i} \\sim \\mathcal{N}(\\mu_i, \\sigma_i^2)$.\n\n#### FLOPs estimation:\n- Computing mean $\\mu_i$ requires ≈2 FLOPs per parameter (sum and divide).\n- Calculating variance $\\sigma_i^2$ requires ≈4 FLOPs per parameter (subtractions, squarings, sum, and divide).\n- Sampling from $\\mathcal{N}(\\mu_i, \\sigma_i^2)$ requires ≈10 FLOPs per parameter (including random number generation and transformation, e.g., via Box-Muller).\n- Minor operations such as random kernel selection (e.g., generating masks or indices) have a time complexity of $O(|S_l| \\times \\max(n_i))$, which is negligible (<1% of total overhead) in practice.\n\n#### Computation overhead of FedPhoneix\nIn each round, FedPhoenix requires $2M$ FLOPs to compute the mean and $4M$ FLOPs for sampling in the cloud server.\nFor the generation of each local model, FedPhoenix reset $\\theta M$ parameters, and each parameter requires 10 FLOPs. Therefore, FedPhoenix requires $K \\times 10 \\theta M$ FLOPs for local model generation.\nIn addition, the model aggregation requires $KM$ FLOPs.\nOverall, in each round, FedPhoenix requires $(6 + (10 \\theta+1)K)M$ FLOPs. Note that the computation overhead of the reset operation is similar to that of the aggregation operation, and all the reset operations are performed in the cloud server. Due to the powerful computing resources of cloud servers, the additional computation overhead of the reset operation is tolerable.\n\nAdditionally, we analyze the computational overhead of the baselines.\nFor **FedProx**, assume that $P$ is the number of local update steps per client; its additional computation overhead is $3 P \\times K \\times M$ FLOPs.\nFor **FedMut**, the additional computation overhead is $M(1 + 2K)$ FLOPs.\nFor **ClusteredSampling**, the additional computation overhead is $3 K N M$ $FLOPs, where $N$ is the total number of clients.\nTherefore, the additional computation overhead of FedPhoneix is similar to the baselines.\n\n\n### Answer to Q3/W3 (Reset for full-connected layers):\n\nSorry for the confusion.\nAs mentioned in the motivation of our paper, we aim to adopt parameter reset to guide the model to learn more generalized features.\nTypically, since CNN-based models employ convolutional layers to construct feature extractors, we prefer to reset the parameters of these convolutional layers.\nNote that, based on the inefficient feature extractor, it is difficult to improve the model performance by simply resetting the fully connected layer (i.e., the classifier).\nWe conducted experiments on Cifar-10 using the VGG model to evaluate the effectiveness of resetting fully connected (FC) layers.\nThe experimental results are as follows:\n| Model | Setting | Conv_Fc      | Only Conv    | Only Fc      | FedAvg       |\n|-------|---------|--------------|--------------|--------------|--------------|\n| VGG   | 0.3     | 83.32 ± 0.18 | 81.46 ± 0.70 | 76.81 ± 0.27 | 75.92 ± 1.16 |\n| VGG   | 0.6     | 83.84 ± 0.58 | 83.24 ± 0.35 | 78.46 ± 0.77 | 79.17 ± 0.06 |\n| VGG   | IID     | 87.18 ± 0.04 | 87.17 ± 0    | 79 ± 0.02    | 80.85 ± 0.01 |\n\nWe can observe that only resetting FC cannot effectively improve the inference accuracy, but resetting both the convolutional layers and the fully connected layers can achieve the best performance.\n\n### Answer to Q4/W4 (Pravicy):\nThanks for your suggestion. Note that the **primary objective of FedPhoenix is to enhance the generalization and robustness of the model**, which cannot provide the formal (ε, δ)-guarantee in DP.\nSince FedPhoenix resets part of the parameters, attackers must make more effort to apply model inversion attacks.\nIn addition, since the parameter reset is based solely on the aggregated global model, FedPhoneix supports secure aggregation. Furthermore, enabling us to handle the model reset operation at the client end will result in better protection against malicious server model reverse attacks. This is because each activated client will have a different training starting point. From the analysis of the computational cost of AQ2, it can also be seen that the cost of our Reset operation is relatively small, and each client can fully bear the cost of resetting their model."}, {"review_id": "0qfdCSu1YH", "reviewer": "Reviewer_QPnF", "summary": "This paper presents a novel federated learning algorithm, named FedPhoneix, which randomly resets partial parameters to guide the FL to learn more generalized features rather than overfitting features.", "questions": "Please refer to the weaknesses", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n* The idea of the paper is interesting and novel. The method design fits the federated learning framework rather than simply migrating existing optimization methods.\n\n* I appreciate the explanation and pre-study in the motivation section of the paper, which exhibits the reasonability of the proposed method.\n\n* The author provides convergence proofs, source code, and related documentation, further improving the soundness of the proposed method.\n\n* The experiments of the paper are comprehensive, which evaluate various factors, especially exploring the adaptability with existing overfitting solutions (e.g., Dropout) and visualizing the model generalization in the appendix. The experimental results show the effectiveness of the proposed method.\n\n## Weakness\n\n* The author should analyze the communication and computational overhead of the method and the baselines. The author only mentions them in the appendix. In my opinion, the author should analyze them more detailed, especially quantitative comparison with baselines. In addition, it is unreasonable to put the discussion of communication and computational overhead in the limitations section.\n\n* The author needs to modify the layout of the paper. For example, Fig 3 covers line 153 and the figure of the framework (Fig 4) and Table 2 is too small. It seems that there is still some space in the paper. The author should adjust the size of tables and figures to make full use of the space. \n\n* The expression of some parts needs to improve. Although the overall writing of the paper is good, the author still needs to revise the expression of some sections to make it easier for readers to understand. For example, Table 1 should appear in the paragraph in which it is mentioned, and it would be better to explain Table 1 in more detail. In addition, in the discussion section at appendix, limitations, communication overhead, computation overhead, and future work should be discussed separately", "was_revised": true, "final_justification": "The authors have resolved the computational overhead issues I raised, and I am satisfied with their clarifications. I am willing to increase my score.", "rebuttal": "### AQ1:Analysis of Server-Side Computational Overhead of the Parameter Reset Mechanism\n\nSorry for the confusion. We conducted a quantitative analysis of the FLOPs (Floating-Point Operations) for each round of the Reset operation. \n\n#### Notations \n-\t$M$ denotes the total number of convolutional layer parameters;\n-\t$K$ denotes the number of activated clients, $|S_l|$ denotes the total number of convolutional layers;\n-\t$\\theta \\in (0,1)$ indicates the perturbation ratio, where $\\lfloor \\theta n_i \\rfloor$ kernels are reset per layer (with $n_i$ kernels in layer $l_i$)\n-\tFor each layer $l_i$, $\\mu_i = \\mathbb{E}[W_{l_i}]$ and $\\sigma_i^2 = \\text{Var}(W_{l_i})$ are computed from the current layer parameters;\n-\tReset kernels are sampled from $\\Delta_{l_i} \\sim \\mathcal{N}(\\mu_i, \\sigma_i^2)$.\n\n#### FLOPs estimation:\n- Computing mean $\\mu_i$ requires ≈2 FLOPs per parameter (sum and divide).\n- Computing variance $\\sigma_i^2$ requires ≈4 FLOPs per parameter (subtractions, squarings, sum, and divide).\n- Sampling from $\\mathcal{N}(\\mu_i, \\sigma_i^2)$ requires ≈10 FLOPs per parameter (including random number generation and transformation, e.g., via Box-Muller).\n- Minor operations like random kernel selection (e.g., generating masks or indices) are $O(|S_l| \\times \\max(n_i))$ and negligible (<1% of total overhead).\n\n#### Computation overhead of FedPhoneix\nIn each round, FedPhoenix requires $2M$ FLOPs to compute the mean and $4M$ FLOPs for sampling in the cloud server.\nFor the generation of each local model, FedPhoenix reset $\\theta M$ parameters, and each parameter requires 10 FLOPs. Therefore, FedPhoenix requires $K \\times 10 \\theta M$ FLOPs for local model generation.\nIn addition, the model aggregation requires $KM$ FLOPs.\nOverall, in each round, FedPhoenix requires $(6 + (10 \\theta+1)K)M$ FLOPs. Note that the computation overhead of the reset operation is similar to that of the aggregation operation, and all the reset operations are performed in the cloud server. Due to the powerful computing resources of cloud servers, the additional computation overhead of the reset operation is tolerable.\n\nAdditionally, we analyze the computational overhead of the baselines.\nFor **FedProx**, assume that $P$ is the number of local update steps per client; its additional computation overhead is $ 3P \\times K \\times M$ FLOPs.\nFor **FedMut**, the additional computation overhead is $M(1 + 2K)$ FLOPs.\nFor **ClusteredSampling**, the additional computation overhead is $3 K N M$ $FLOPs, where $N$ is the total number of clients.\nTherefore, the additional computation overhead of FedPhoneix is similar to the baselines.\n\n### AQ: About writing optimization\n\nThank you for your rich writing experience. We will make adjustments to Figure 3 and Figure 4 in the revised version. And further optimize the presentation of the core concepts (such as Table 1). In the discussion section, we will refine the framework to make writing more coherent and organized."}, {"review_id": "vcutccmtmT", "reviewer": "Reviewer_iGvs", "summary": "This work proposes a new FL framework called PedPhoenix to address the issue that the global model in FL\ncan be easily stuck in local optima due to its average aggregation strategy. It considers a stochastic reset\nstrategy for parameters in the global model at each round to learn generalized features more effectively.", "questions": "1. In the proposed dynamic stabilization strategy, \nthe parameter reset depends on the location of convolution layers because the criterion depends on i \\times r_s\nwhere i is the index of the convolution layer. This implies that the convolution layer with small i has less chance\nto reset than the convolution layer with large i. As the communication round r increases, this becomes more severe.\nIs there any reason why the criterion i \\times r_s depends on the index(location in the model) of the layers? \nIs it related with the fact that different convolutional layers learn different features (low-level, mid-level, high-level) \ndepending on their locations in the model?\n\n2. For reset, the mean and variance of the current layer parameters are used in sampling, but the kernel contains\nimportant feature information and each parameter contains different information. By replacing sampled values\nthe relative importance of each parameter seems to be disregarded. From this perspective,\nis it a good strategy to replace all the parameter values by the sampled values? What happens if we select a portion\nof parameters with, e.g., large values and reset them by the sampled values? The parameters with very small values \ncan be disregarded even in the reset because their relative importance is not significant.\n\n3. When some features are not general but specific to some clients, its training is not conducted frequently due to\nrandom selection of active clients and furthermore even when its training is conducted, by aggregating the weights for an \nupdated global model its impact becomes less significant. This seems to be true when a large number of clients involve \nin FL, which is a usual setting in FL. \nEven though nice experimental results in Fig. 2 and Fig. 3 are provided, the above issue needs more detailed discussion.\n\n4. It seems that a smaller value of theta (reset proportion) is used for a more heterogeneous case (beta = 0.3, nonIID)\nin section 6.1. It seems that a more heterogeneous case needs more reset portion. Is there a good reason for it?\n\n5. In algorithm 2, the notations of the reset proportion are different in the Reset function (mu_0) and in line 7 (theta). \nIn the context both notations are also used to denote the reset proportion. It is better to use the same notation for a parameter.\n\n6. In ablation study, a comparison study with Kaiming-Normal is conducted. It would be helpful if more comparison studies \nwith other distributions are provided because reset is the key idea and it would be interesting if the use of other \ndistributions for reset sampling can affect the results.\n\n7.For non IID case, when local datasets have different features, most of which are not shared, fitting to local\ndatasets would rather improve the accuracy because of collaboration with which each client can learn the features that\ncannot be trained with its local dataset. For instance, the input space is partitioned and each local dataset\nis generated from each subset in the partition. Then, does the proposed algorithm still perform well enough?", "limitations": "The proposed algorithm focuses on FL with convolutional neural networks and hence can be applied to limited\nscenarios. A generalization of the proposed approach to other model architectures needs to be discussed.\nModel homogeneity is inevitable in the proposed model, which is another limitation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The key idea of this work is to learn more generalized features than client-specific features by stochastically\nresetting a few parameters in the global model during communication rounds.\nIn the proposed approach, copies (with reset) of the global model are downloaded to active clients, meaning that\nmodel homogeneity is crucial. This is a limitation. Moreover, since the global model is usually large-scaled, \ndownloading and uploading the copies usually require high cost.", "was_revised": true, "final_justification": "I have carefully reviewed the responses. My concerns on the reset strategy and sampling distribution are well addressed and the other concerns have been also addressed. So, I am happy to raise my score to 4.", "rebuttal": "### Answer to Q1\nSorry for the confusion. Since the reasoning of deep layers in neural networks is based on the output from shallow layers, we prioritize reducing the reset rate of shallow layers to ensure training stability.\nFrom the perspective of feature levels, as established by seminal works such as **Zeiler & Fergus (2014, \"Visualizing and Understanding Convolutional Networks\")**, shallow layers learn general-purpose, low-level features (e.g., edges, textures) that stabilize quickly, and high-level features are extracted from these low-level features. Therefore, we first reduce the reset rate of shallow layers to stabilize the learned low-level features and facilitate the learning of high-level features based on stable low-level features.\n\n### Answer to Q2\nThanks for your insightful suggestion.\n\nOur primary motivation for resetting the *entire* kernel is to treat it as a single feature extraction unit. By completely re-initializing it, we aim to force the model to explore entirely new feature patterns, thereby enhancing generalization and feature diversity, which is the core goal of FedPhoenix.\n\nWe conducted new ablation studies where we selectively reset only a portion of the parameters within a kernel based on their magnitude (top 1/3 and top 2/3). We conducted these experiments by keeping the kernel reset ratio ($\\theta$) constant and by increasing it to maintain a constant total number of reset parameters. The experimental results are as follows:\n\n| Setting | Top1/3       | Top2/3       | Top1/3 Fixed θ | Top2/3 Fixed θ | FedPhoenix   |\n| ------- | ------------ | ------------ | -------------- | -------------- | ------------ |\n| 0.3     | 77.2 ± 1.87  | 78.47 ± 1.32 | 76.86 ± 0.98   | 79.55 ± 0.95   | 80.12 ± 0.54 |\n| 0.6     | 80.68 ± 0.17 | 80.9 ± 1.06  | 80.35 ± 0.05   | 81.76 ± 0.78   | 82.28 ± 0.82 |\n| IID     | 83.01 ± 0.01 | 84.78 ± 0.02 | 84.26 ± 0.01   | 84.63 ± 0.01   | 84.85 ± 0.07 |\n\n\nThe results show that compared with the new plan, the original plan performs better. This is mainly because partial resets disrupt the learned structure of a kernel, potentially creating gradient conflicts during training. The holistic reset of the entire unit appears to be more effective for our purpose.\n\n\n\n### Answer to Q3\nSorry for the confusion. From a geometric perspective, despite limited participation of each client, learning non-generalized features will still cause the model to be stuck in a local optimum, i.e., a sharp area. Please note that due to the similarities in the characteristics of different data, in this area, the local optimum of different clients will be close but different. As the training progresses, the optimization directions of different clients will conflict, making it difficult to reach a consensus and thus getting stuck in the local optimum.\nParameter reset can enable the model to directly jump out of the local optimal area and find a new solution. Please note that the generalization area is relatively flat. Resetting part of the model's parameters is not enough to cause the aggregated model to jump out of this area.\n\n### Answer to Q4\nSorry for the confusion. The optimal perturbation ratio is determined by a delicate trade-off between **promoting exploration** and **preserving the stability of learned representations**, and this balance shifts with the degree of data heterogeneity.\nIncreased data heterogeneity will lead to greater differences in local model optimization directions, making it challenging to reach a consensus on the optimal direction during training.\nIn these scenarios, resetting more parameters will seriously increase the time overhead of FL.\n\n\n### Answer to Q5\nSorry for the confusion. We will unify the notations in the final version.\n\n\n### Answer to Q6\nThanks for your suggestion. We conduct a comprehensive ablation study to investigate the impact of various distributions on the performance of our proposed method. We expanded our experiments to include several widely recognized and statistically diverse distributions:\n\n- **Xavier/Glorot (Normal & Uniform):** To validate the effectiveness of our reset mechanism with another mainstream initialization standard.\n- **Truncated Normal:** To evaluate a more conservative reset strategy that enhances stability by limiting extreme sample values.\n- **Orthogonal:** To test whether introducing \"structured\" randomness, which is known to preserve gradient norms, offers advantages over purely random sampling.\n- **Laplace:** To explore the effect of a distribution with different statistical properties (more peaked and promoting sparsity).\n\nFor a fair comparison, all these distributions were implemented using the \"adaptive\" strategy, i.e., they were scaled using the mean and variance of the current layer's weights, consistent with our proposed `Adp_Normal` method. The experiments were conducted on the CIFAR-10 dataset and the ResNet-18 model.\n\n| Non-IID Level |  Adp_Normal  | Adp_Uniform  | Kaiming_Normal | Kaiming_Uniform | Truncated_Normal | Xavier_Normal | Xavier_Uniform | Orthogonal |   Laplace    |\n| :------------ | :----------: | :----------: | :------------: | :-------------: | :--------------: | :-----------: | :------------: | :--------------: | :----------: |\n| **0.3**       | 80.12 ± 0.54 | 80.02 ± 0.41 |  79.34 ± 0.64  |  77.58 ± 2.34   |   79.42 ± 1.45   | 79.47 ± 0.59  |  78.87 ± 2.01  | 80.00 ± 0.16 | 78.46 ± 0.34 |\n| **0.6**       | 82.28 ± 0.82 | 81.86 ± 0.24 |  79.97 ± 0.17  |  80.34 ± 0.41   |   81.70 ± 0.09   | 80.60 ± 0.21  |  81.03 ± 0.60  | 82.68 ± 0.11 | 81.74 ± 0.07 |\n| **IID**       | 85.06 ± 0.03 | 85.25 ± 0.04 |  84.70 ± 0.01  |  84.52 ± 0.03   |   85.08 ± 0.02   | 85.03 ± 0.02  |  84.75 ± 0.01  | 85.42 ± 0.03 | 85.00 ± 0.02 |\n\nBased on these results, we can find that the core strength of our method lies in the adaptive reset strategy itself, rather than the specific choice of distribution. The consistently high performance across diverse distributions (Normal, Uniform, Orthogonal, etc.) demonstrates the mechanism's robustness and low sensitivity. While the Orthogonal distribution shows a slight advantage in some cases, our proposed Adp_Normal is highly competitive across all scenarios. This confirms that adapting the perturbation to the layer's current statistics is the key factor for success, making our approach broadly effective.\n\n\n\n### Answer to Q7\nThanks for your suggestion. We conducted a new experiment on the **Office-31 dataset**, a standard benchmark for cross-domain scenarios, where the feature space of each client is quite different. We partitioned the dataset by its three distinct domains (**Amazon, DSLR, Webcam**), assigning each domain to a separate client. The experimental results are as follows:\n\n| Dataset   | FedAvg       | FedProx      | FedMut       | CluSamp      | FedGen       | FedPhoenix   |\n| --------- | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Office_31 | 47.84 ± 1.78 | 48.12 ± 1.68 | 43.99 ± 1.59 | 46.42 ± 1.83 | 49.13 ± 2.26 | 54.17 ± 0.96 |\n\nAs shown in the table, our method achieves a **significant performance gain** over the baselines. This demonstrates that our approach not only performs well but also excels in this challenging, feature-skewed environment. By mitigating client drift and encouraging the exploration of a shared feature space, our method facilitates more effective collaboration, leading to a more robust and generalized global model."}, {"review_id": "y4K9sgedvG", "reviewer": "Reviewer_NLhY", "summary": "This paper proposes the FedPhoenix framework to address the poor generalization ability of global models. The core idea is to randomly reset a subset of parameters to disrupt certain characteristics of the global model.", "questions": "See the weaknesses", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method is concise, simple, and effective.\n\n2. The authors provide code (in the supplementary materials).\n\n3. There is substantial theoretical analysis.\n\n4. The experiments are thorough, conducted on three different models and three datasets — notably including ImageNet. The ablation study is also well-designed.\n\nWeaknesses:\n\nMajor:\n1. The related work section is not well written. The author categorizes the FL methods in the introduction, which looks good, but does not provide a detailed introduction and discussion in the related work section. I suggest the author discuss more related work.\n\n2.The experiments should include more extreme scenarios, such as further reducing the beta value (0.1, 0.01).\n\n3.The paper mentions using more computational resources — this is understandable, but it would be helpful to include some quantitative data on that.\n\nMinor:\n\nFigure 1 is too small. It seems that there is more space on page 9. The author should enlarge Figure 1.", "was_revised": true, "final_justification": "I just carefully reviewed your responses to all the reviewers’ comments. My concerns have been largely addressed, and it seems the other reviewers’ issues have also been resolved. Please make sure the final version fills the full 9 pages. I will raise my score to 5. Thanks for your rebuttal.", "rebuttal": "### AQ1 regarding the related work：\n\nThank you for your suggestion. We will provide a more detailed introduction to the related work in the revised version.\n\n\n\n### AQ2:\n\nThank you for your suggestion. Adding comparisons regarding extremely heterogeneous data will further highlight the advantages of our method. The following table results show that our method still achieved the best outcome compared to the baseline.\n\n| Model     | Setting | FedAvg      | FedProx     | FedMut      | CluSamp     | FedGen      | FedPhoenix  |\n|-----------|---------|-------------|-------------|-------------|-------------|-------------|-------------|\n| ResNet-18 | 0.01   | 29.33 ± 0.30 | 27.88 ± 0.41 | 28.20 ± 0.33 | 28.53 ± 0.26 | 29.46 ± 0.04 | 35.80 ± 0.93 |\n|  | 0.1    | 45.48 ± 0.03 | 45.41 ± 0.16 | 47.55 ± 0.14 | 45.56 ± 0.36 | 45.29 ± 7.33 | 55.80 ± 0.11 |\n| VGG16     | 0.01   | 32.24 ± 0.16 | 31.79 ± 0.37 | 32.09 ± 0.16 | 30.74 ± 0.57 | 32.03 ± 1.89 | 33.10 ± 0.41 |\n|     | 0.1    | 47.75 ± 0.11 | 48.26 ± 0.18 | 49.59 ± 0.07 | 47.72 ± 0.05 | 49.30 ± 0.29 | 52.86 ± 0.14 |\n| MobNet-V1 | 0.01   | 19.87 ± 0.31 | 19.88 ± 0.11 | 21.48 ± 0.15 | 18.31 ± 0.09 | 21.01 ± 0.14 | 27.56 ± 0.57 |\n| | 0.1    | 36.59 ± 0.18 | 36.86 ± 0.03 | 39.55 ± 0.15 | 35.48 ± 0.06 | 37.93 ± 0.13 | 50.36 ± 0.11 |\n\n\n\n### Answer to Q3 (Analysis of Computational Overhead):\nSorry for the confusion. We conduct a quantitative analysis of the FLOPs (Floating-Point Operations) for each round of the Reset operation. \n\n#### Notations \n-\t$M$ denotes the total number of convolutional layer parameters;\n-\t$K$ denotes the number of activated clients, $|S_l|$ denotes the total number of convolutional layers;\n-\t$\\theta \\in (0,1)$ indicates the perturbation ratio, where $\\lfloor \\theta n_i \\rfloor$ kernels are reset per layer (with $n_i$ kernels in layer $l_i$)\n-\tFor each layer $l_i$, $\\mu_i = \\mathbb{E}[W_{l_i}]$ and $\\sigma_i^2 = \\text{Var}(W_{l_i})$ are computed from the current layer parameters;\n-\tThe reset kernels are sampled from $\\Delta_{l_i} \\sim \\mathcal{N}(\\mu_i, \\sigma_i^2)$.\n\n#### FLOPs estimation:\n- Computing mean $\\mu_i$ requires ≈2 FLOPs per parameter (sum and divide).\n- Calculating variance $\\sigma_i^2$ requires ≈4 FLOPs per parameter (subtractions, squarings, sum, and divide).\n- Sampling from $\\mathcal{N}(\\mu_i, \\sigma_i^2)$ requires ≈10 FLOPs per parameter (including random number generation and transformation, e.g., via Box-Muller).\n- Minor operations like random kernel selection (e.g., generating masks or indices) are $O(|S_l| \\times \\max(n_i))$ and negligible (<1% of total overhead).\n\n#### Computation overhead of FedPhoneix\nIn each round, FedPhoenix requires $2M$ FLOPs to compute the mean and $4M$ FLOPs for sampling in the cloud server.\nFor the generation of each local model, FedPhoenix reset $\\theta M$ parameters, and each parameter requires 10 FLOPs. Therefore, FedPhoenix requires $K \\times 10 \\theta M$ FLOPs for local model generation.\nIn addition, the model aggregation requires $KM$ FLOPs.\nOverall, in each round, FedPhoenix requires $(6 + (10 \\theta+1)K)M$ FLOPs. Note that the computation overhead of the reset operation is similar to that of the aggregation operation, and all the reset operations are performed in the cloud server. Due to the powerful computing resources of cloud servers, the additional computation overhead of the reset operation is tolerable.\n\nAdditionally, we analyze the computational overhead of the baselines.\nFor **FedProx**, assume that $P$ is the number of local update steps per client; its additional computation overhead is $3 P \\times K \\times M$ FLOPs.\nFor **FedMut**, the additional computation overhead is $M(1 + 2K)$ FLOPs.\nFor **ClusteredSampling**, its additional computation overhead is $3 K N M$ FLOPs, where $N$ is the total number of clients.\nTherefore, the additional computation overhead of FedPhoneix is similar to the baselines.\n\n### AQ4：\n\nThank you for your suggestion. We will adjust the page layout in the revised version."}]

Author rebuttal:

Meta-review: The work introduces FedPhoenix, a novel FL framework designed to mitigate overfitting in non-IID settings through a server-side dynamic parameter reset mechanism.
 
The primary strengths of this submission are its novelty and soundness. Reviewers consistently praised the central idea as interesting, innovative, and well-suited to the federated learning paradigm, avoiding common implementation pitfalls like increased communication overhead or the need for public proxy data. The authors provide a clear motivation for their approach, theoretical analysis for convergence, and comprehensive empirical evaluations that convincingly demonstrate significant accuracy improvements over state-of-the-art methods. 
 
Furthermore, the authors engaged constructively during the rebuttal period, providing new experiments and clarifications that addressed the majority of reviewers' concerns, leading to score increases from three of the four reviewers. All reviewers ultimately agreed that their main issues were resolved. 

Nevertheless, here are several areas for improvement that should be addressed in the final camera-ready version. Firstly, the most significant limitation noted by reviewers is the empirical scope; experiments are confined to CNNs, while validation on more modern architectures like Transformers is missing. The final paper would be substantially strengthened by including at least preliminary Transformer experimental results or by clearly positioning this as a critical avenue for future work. Secondly,  reviewers noted that the analysis of computational overhead could be more detailed and needs better visibility. Finally, the paper should include a more thorough discussion on the sensitivity to hyperparameters like reset rates and rounds, which can be included in the supplement. 

In summary, this is a technically solid paper with a novel contribution and strong results. The reasons to accept decidedly outweigh the weaknesses, and incorporating the suggested revisions will further enhance the quality and impact of this work.
Decision: Accept (poster)