Title: Soft-consensual Federated Learning for Data Heterogeneity via Multiple Paths
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated learning

data heterogeneity

soft consensus

multiple-solving paths
Abstract: Federated learning enables collaborative training while preserving the privacy of all participants. However, the heterogeneity in data distribution across multiple training nodes poses significant challenges to the construction of federated models. Prior studies were dedicated to mitigating the effects of data heterogeneity by using global information as a blueprint and restricting the local update of the model for reaching a "hard consensus". But this practice makes it difficult to balance local and global information, and it neglects to negotiate amicably between local and global models to reach mutually agreeable results, called ``soft consensus". In this paper, a multiple-path solving method is proposed to balance global and local features and combine these two feature preference paths to reach a soft consensus. Rather than relying on global information as the sole criterion, a negotiation process is employed to address the same objective by accommodating diverse feature preferences, thereby facilitating the discovery of a more plausible solution through multiple distinct pathways. Considering the overwhelming power of local features during local training, a swapping strategy is applied to weaken them to balance the solution paths. Moreover, to minimize the additional communication cost caused by the introduction of multiple paths, the solution of the task network is converted into data adaptation to reduce the amount of parameter transmission. Extensive experiments are conducted to demonstrate the advantages of the proposed method.


Peer reviews:
comments: SUMMARY:
The authors propose a multi-path neural network optimization method from the perspective of global and local soft consensus in federated learning, combining global and local paths to seek their consensus solution. The authors also use swap strategy and data adaptation method to further reduce additional communication overhead. The effectiveness of the proposed method is validated through extensive experiments.

STRENGTHS AND WEAKNESSES:
Strengths

a. The idea of using multi-paths to find a consensus solution is insightful.

b. The motivation statement is clear, and the paper provides a detailed explanation of the differences between multi-paths and other similar methods.

c. The paper validates the effectiveness of the proposed framework through a variety of experiments. 

Weaknesses

a. The multi-path method appears to introduce additional computational overhead.

b. Balancing different paths introduces additional hyperparameter, and there is no related hyperparameter analysis.

c. The swap strategy may lead to additional information leakage.

QUESTIONS:
a. Does the proposed multi-path method require two identical networks to simulate different solution paths? Would this introduce additional network parameters? Could this lead to an unfair comparison when contrasting with other methods?

b. How do you balance global and local paths? Is there any related hyperparameter analysis?

c. What does Figure 9 mean? Please clarify further.

d. What does the binding and swapping in Figure 2 specifically represent? I did not find an explanation for this binding operation in the paper.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a federated learning method based on multi-path solution, which seeks a consensus between global feature preferences and local feature preferences through a multi-path approach, avoiding the hard consensus typically achieved by federated learning methods for heterogeneous data. The method balances and jointly optimizes global and local models to construct the aggregated model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper describes an interesting multi-path solution method. Negotiating between global and local preferences to reach a consensus appears to be an effective solution.\n2. The authors provide theoretical proofs in paper.\n3. Extensive experiments demonstrate the effectiveness of the proposed method, including comparative studies to showcase the superiority of the multi-path method.\n\nWeaknesses\n1. Sections 2.3 and 2.4 contain excessive mathematical notation without a summary table clarifying their meanings.\n2. The methodology for data adaptation is not sufficiently elaborated.\n3. The discussion lacks an in-depth analysis of the balance between the two paths. Some explanations are unclear, for instance, the relevance of Equations (6) and (7) to the multi-path method is not explain well.\n\nQUESTIONS:\n1. Please respond to what I mentioned in my weaknesses. \n2. Could you further clarify the derivations and motivations behind Equations (3)–(7)? While the supplementary material (Section B) effectively highlights the novelty of the multi-path approach, the main text requires more detailed explanations for these Equations.\n3. As noted in the weaknesses, there is no discussion on the balance between the two paths. I observed hyperparameters in the multi-path optimization, but the paper lacks a sensitivity analysis on these parameters. Could you address this?\n4. In Table 2, the performance appears to degrade when using the multi-path method, which contradicts the conclusions drawn in Figure 3. What explains this discrepancy?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the data heterogeneity challenge in federated learning, this paper proposes a FedMP framework. This method optimizes multiple solution paths that prefer global and local features and allows them to negotiate to produce a better solution. To balance the path strength, the paper introduces a swapping strategy and uses data adaptation to reduce communication overhead. Comprehensive experiments show that this method outperforms existing advanced methods under heterogeneous data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The article is well-structured, the problem and motivation are clearly described, and the proposed method is easy to understand.\n2. The proposed method is innovative, and it forms a complete solution by coordinating multiple paths with different feature preferences through a shared projection layer, combined with swapping strategies and data adaptation.\n3. The authors provide comprehensive experimental results to demonstrate the effectiveness of the method and each module.\n\nWeaknesses\n1. In local preference swapping strategy, the local model of the client is not the model trained locally in the previous round, but a local model trained by another client randomly downloaded from the server. Although ablation experiments prove its effectiveness, please explain its rationality, which seems counterintuitive. Besides, the local model of client A may come from client B in the previous round. The paper claims that this path represents \"local feature preference\". However, for client A, it is actually using the local preference model of client B. Directly describing the preferences of other clients as local preferences may cause ambiguity.\n2. In multiple paths data adaptation, to reduce communication costs, FedMP keeps the full task network constant and transfers the optimization objective from a large network to a data adapter. Will this limit the optimal model performance?\n3. I would like to know what loss function L_p used in the experiment\n\nQUESTIONS:\nI suggest that the authors provide the fundamental rationale and conceptual clarity of the \"local preference swapping strategy\", analyze the performance bottlenecks that may be caused by data adaptation, and the choice of loss function. For details, see the previous \"Weaknesses\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

comments: SUMMARY:
A multi-path solution method is proposed to combine global and local features in federated learning. This method achieves soft consensus among multiple paths, avoiding the previous approaches of using global information as the sole criterion. Considering the dominance of local features in local training, the authors employ an swapping strategy to weaken the local solution path. They adopt a data-adaptive approach to reduce the amount of model parameter transmission, thereby minimizing additional communication overhead.

STRENGTHS AND WEAKNESSES:
Strengths

i. The proposed multi-path solution method is interesting, and experiments are conducted to demonstrate its feasibility.

ii. A case study is provided at the beginning to justify the necessity of the proposed method.

iii. The authors explain the multi-path method to distinguish it from other methods.

Weaknesses

i. The validation experiments seem to be limited to image classification, lacking other types of tasks.

ii. The paper lacks hyperparameter analysis, such as analysis of path balancing.

iii. Some descriptions in the paper are unclear and require further refinement.

QUESTIONS:
i. Can the proposed method be applied to other models or tasks beyond image classification?

ii. Does the multi-path method employed in the paper enhance the final model's capabilities by increasing the number of model parameters?  

iii. How is the data adaptation method described in the paper specifically implemented to reduce model communication overhead?  

iv. If two networks are trained together under the same loss function, can this be considered multi-path learning?  

v. Is the proposed multi-path learning scheme unrelated to federated learning?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02