Title: Rising from Ashes: Generalized Federated Learning via Dynamic Parameter Reset
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning
Abstract: Although Federated Learning (FL) is promising in privacy-preserving collaborative model training, it faces low inference performance due to heterogeneous data among clients. 
Due to heterogeneous data in each client, FL training easily learns the specific overfitting features.
Existing FL methods adopt the coarse-grained average aggregation strategy, which causes the global model to easily get stuck in local optima, resulting in low generalization of the global model.
Specifically, this paper presents a novel FL framework named FedPhoenix to address this issue, which stochastically resets partial parameters to destroy some features of the global model in each round to guide the FL training to learn multiple generalized features for inference rather than specific overfitting features.
Experimental results on various well-known datasets demonstrate that compared to SOTA FL methods, FedPhoenix can achieve up to 20.73\% accuracy improvement.


Peer reviews:
comments: SUMMARY:
This paper proposes the FedPhoenix framework to address the poor generalization ability of global models. The core idea is to randomly reset a subset of parameters to disrupt certain characteristics of the global model.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The method is concise, simple, and effective.

2. The authors provide code (in the supplementary materials).

3. There is substantial theoretical analysis.

4. The experiments are thorough, conducted on three different models and three datasets — notably including ImageNet. The ablation study is also well-designed.

Weaknesses:

Major:
1. The related work section is not well written. The author categorizes the FL methods in the introduction, which looks good, but does not provide a detailed introduction and discussion in the related work section. I suggest the author discuss more related work.

2.The experiments should include more extreme scenarios, such as further reducing the beta value (0.1, 0.01).

3.The paper mentions using more computational resources — this is understandable, but it would be helpful to include some quantitative data on that.

Minor:

Figure 1 is too small. It seems that there is more space on page 9. The author should enlarge Figure 1.

QUESTIONS:
See the weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new FL framework called PedPhoenix to address the issue that the global model in FL\ncan be easily stuck in local optima due to its average aggregation strategy. It considers a stochastic reset\nstrategy for parameters in the global model at each round to learn generalized features more effectively.\n\nSTRENGTHS AND WEAKNESSES:\nThe key idea of this work is to learn more generalized features than client-specific features by stochastically\nresetting a few parameters in the global model during communication rounds.\nIn the proposed approach, copies (with reset) of the global model are downloaded to active clients, meaning that\nmodel homogeneity is crucial. This is a limitation. Moreover, since the global model is usually large-scaled, \ndownloading and uploading the copies usually require high cost.\n\nQUESTIONS:\n1. In the proposed dynamic stabilization strategy, \nthe parameter reset depends on the location of convolution layers because the criterion depends on i \\times r_s\nwhere i is the index of the convolution layer. This implies that the convolution layer with small i has less chance\nto reset than the convolution layer with large i. As the communication round r increases, this becomes more severe.\nIs there any reason why the criterion i \\times r_s depends on the index(location in the model) of the layers? \nIs it related with the fact that different convolutional layers learn different features (low-level, mid-level, high-level) \ndepending on their locations in the model?\n\n2. For reset, the mean and variance of the current layer parameters are used in sampling, but the kernel contains\nimportant feature information and each parameter contains different information. By replacing sampled values\nthe relative importance of each parameter seems to be disregarded. From this perspective,\nis it a good strategy to replace all the parameter values by the sampled values? What happens if we select a portion\nof parameters with, e.g., large values and reset them by the sampled values? The parameters with very small values \ncan be disregarded even in the reset because their relative importance is not significant.\n\n3. When some features are not general but specific to some clients, its training is not conducted frequently due to\nrandom selection of active clients and furthermore even when its training is conducted, by aggregating the weights for an \nupdated global model its impact becomes less significant. This seems to be true when a large number of clients involve \nin FL, which is a usual setting in FL. \nEven though nice experimental results in Fig. 2 and Fig. 3 are provided, the above issue needs more detailed discussion.\n\n4. It seems that a smaller value of theta (reset proportion) is used for a more heterogeneous case (beta = 0.3, nonIID)\nin section 6.1. It seems that a more heterogeneous case needs more reset portion. Is there a good reason for it?\n\n5. In algorithm 2, the notations of the reset proportion are different in the Reset function (mu_0) and in line 7 (theta). \nIn the context both notations are also used to denote the reset proportion. It is better to use the same notation for a parameter.\n\n6. In ablation study, a comparison study with Kaiming-Normal is conducted. It would be helpful if more comparison studies \nwith other distributions are provided because reset is the key idea and it would be interesting if the use of other \ndistributions for reset sampling can affect the results.\n\n7.For non IID case, when local datasets have different features, most of which are not shared, fitting to local\ndatasets would rather improve the accuracy because of collaboration with which each client can learn the features that\ncannot be trained with its local dataset. For instance, the input space is partitioned and each local dataset\nis generated from each subset in the partition. Then, does the proposed algorithm still perform well enough?\n\nLIMITATIONS:\nThe proposed algorithm focuses on FL with convolutional neural networks and hence can be applied to limited\nscenarios. A generalization of the proposed approach to other model architectures needs to be discussed.\nModel homogeneity is inevitable in the proposed model, which is another limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel federated learning algorithm, named FedPhoneix, which randomly resets partial parameters to guide the FL to learn more generalized features rather than overfitting features.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n* The idea of the paper is interesting and novel. The method design fits the federated learning framework rather than simply migrating existing optimization methods.\n\n* I appreciate the explanation and pre-study in the motivation section of the paper, which exhibits the reasonability of the proposed method.\n\n* The author provides convergence proofs, source code, and related documentation, further improving the soundness of the proposed method.\n\n* The experiments of the paper are comprehensive, which evaluate various factors, especially exploring the adaptability with existing overfitting solutions (e.g., Dropout) and visualizing the model generalization in the appendix. The experimental results show the effectiveness of the proposed method.\n\n## Weakness\n\n* The author should analyze the communication and computational overhead of the method and the baselines. The author only mentions them in the appendix. In my opinion, the author should analyze them more detailed, especially quantitative comparison with baselines. In addition, it is unreasonable to put the discussion of communication and computational overhead in the limitations section.\n\n* The author needs to modify the layout of the paper. For example, Fig 3 covers line 153 and the figure of the framework (Fig 4) and Table 2 is too small. It seems that there is still some space in the paper. The author should adjust the size of tables and figures to make full use of the space. \n\n* The expression of some parts needs to improve. Although the overall writing of the paper is good, the author still needs to revise the expression of some sections to make it easier for readers to understand. For example, Table 1 should appear in the paragraph in which it is mentioned, and it would be better to explain Table 1 in more detail. In addition, in the discussion section at appendix, limitations, communication overhead, computation overhead, and future work should be discussed separately\n\nQUESTIONS:\nPlease refer to the weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FedPhoenix, an FL framework designed to enhance model generalization and inference performance in non-IID (non-independent and identically distributed) data scenarios. The core concept of FedPhoenix is a server-side dynamic parameter reset mechanism. This mechanism stochastically resets partial parameters of the global model before dispatching them to clients for local training, aiming to disrupt client-specific overfitting features and encourage the learning of more generalized features across the federated network. The authors contribute a novel FL framework with an adaptive parameter reset strategy, a dynamic stabilization strategy, theoretical analysis for convergence, and comprehensive empirical evaluations demonstrating significant accuracy improvements over state-of-the-art FL methods on various datasets and model architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n**+** The central idea of dynamic parameter reset on the server side is a genuinely innovative approach to combat overfitting in non-IID federated learning.  \n\n**+**  The paper clearly articulates the problem of models overfitting to local client data and provides compelling empirical evidence to demonstrate that existing FedAvg models indeed rely on a narrow set of specific features. \n\n**+** This design avoids the common pitfalls of client-side modifications, additional communication overhead, or the need for public proxy datasets, which often limit the practicality and deployability of other FL debiasing or generalization methods.\n\nAreas for improvement\n\n**-**  Given the increasing prominence of Transformers in FL, empirical validation is exclusively confined to vision tasks and CNNs; this significantly limits the scope of generalization implied.\n\n**-** The authors state that the server-side computational overhead for generating multiple model copies with reset parameters is comparable to FedAvg. However, the detailed quantification of this overhead is not fully convincing. For very large models or a significantly higher number of active clients (K), this overhead could become a practical bottleneck. \n\n**-** Fully connected layers are often critical for final classification and can also be prone to overfitting. The rationale for this specific focus, and whether extending the parameter reset mechanism to these layers would offer further benefits or introduce new challenges, is not thoroughly explored.\n\n**-** While introducing noise can have privacy benefits, this is a weaker claim than formal privacy guarantees (e.g., differential privacy). The paper itself suggests integrating differential privacy in future work. Clarifying the specific type of privacy enhancement offered by the current mechanism and distinguishing it from more formal privacy guarantees would provide a more accurate picture.\n\nQUESTIONS:\n**-** Could you elaborate on the practical implications and potential adaptations of FedPhoenix for Transformer architectures and NLP tasks?\n\nActionable Guidance: Please discuss how the parameter reset mechanism would apply to specific components of Transformers. A conceptual outline of how the destruction and restoration of features would manifest in this domain, and any anticipated challenges or necessary modifications, would be highly valuable.\n\n\n**-** Please provide a more detailed analysis of the server-side computational overhead introduced by the parameter reset mechanism.\nActionable Guidance: Quantify the computational cost (FLOPs or actual wall-clock time) of the Reset function and the overall server-side processing per round. Present this data as a function of model size (number of parameters) and the number of activated clients K to better assess scalability.\n\n\n**-** What is the rationale behind exclusively applying the parameter reset to convolutional layers and not to fully connected layers?\nActionable Guidance: Please provide a justification for this design choice. \n\n\n**-** Could you provide a more precise characterization of the privacy-preserving aspect of FedPhoenix beyond increasing uncertainty and diversity?\nActionable Guidance: Clarify whether the current mechanism offers any formal privacy guarantees against specific attack models like membership inference or model inversion.\n\nLIMITATIONS:\nIn Appendix C.2, the authors acknowledge several limitations. I would argue that some of them should move to the main text. \n\n**-** The ablation studies in Figure 7 show that the performance of FedPhoenix is sensitive to the reset rounds and reset rates hyperparameters. While the paper states that all variants eventually converge, achieving optimal performance appears to depend on careful tuning. In real-world FL settings, where validation data might be limited or difficult to obtain, the sensitivity to these parameters could hint at a practical challenge for deployment.\n\n**-** The core concept relies on \"destroying\" features to encourage generalization. While effective for CNNs, the interpretation, and effectiveness of this \"destruction\" might vary significantly for different types of neural network architectures (RNN for sequential data, GNN for graph data) because of different structural implications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}