Soft-consensual Federated Learning for Data Heterogeneity via Multiple Paths
Authors: Sheng Huang
Lele Fu
Fanghua Ye
Tianchi Liao
Bowen Deng
zhangchuanfu
Chuan Chen
Venue: NeurIPS 2025 poster ()
Keywords: Federated learning
data heterogeneity
soft consensus
multiple-solving paths
Primary area: deep_learning
TL;DR: A soft-consensual FL framework is constructed by a multi-path solving method.

Abstract: Federated learning enables collaborative training while preserving the privacy of all participants. However, the heterogeneity in data distribution across multiple training nodes poses significant challenges to the construction of federated models. Prior studies were dedicated to mitigating the effects of data heterogeneity by using global information as a blueprint and restricting the local update of the model for reaching a "hard consensus". But this practice makes it difficult to balance local and global information, and it neglects to negotiate amicably between local and global models to reach mutually agreeable results, called ``soft consensus". In this paper, a multiple-path solving method is proposed to balance global and local features and combine these two feature preference paths to reach a soft consensus. Rather than relying on global information as the sole criterion, a negotiation process is employed to address the same objective by accommodating diverse feature preferences, thereby facilitating the discovery of a more plausible solution through multiple distinct pathways. Considering the overwhelming power of local features during local training, a swapping strategy is applied to weaken them to balance the solution paths. Moreover, to minimize the additional communication cost caused by the introduction of multiple paths, the solution of the task network is converted into data adaptation to reduce the amount of parameter transmission. Extensive experiments are conducted to demonstrate the advantages of the proposed method.

Soft-consensual Federated Learning for Data Heterogeneity via Multiple Paths
Sheng Huang', Lele Fu', Fanghua Ye², Tianchi Liao', Bowen Deng', Chuanfu Zhang', Chuan Chen
Sun Yat-sen University, Guangzhou, China Tencent Inc., Shenzhen, China
fhuangsh253, fulle, liaotch, dengbw3) | @2a0m22....... sysu..ssuuuuuu cn (zhangchf9, chenchuan] @mail. sysu, cn fanghua. ye 211gmaill... com
Abstract
Federated learning enables collaborative training while preserving the privacy of all participants. However, the heterogeneity in data distribution across multiple training nodes poses significant challenges to the construction of federated models. Prior studies were dedicated to mitigating the effects of data heterogeneity by using global information
blueprint and restricting the local update of the model for reaching a "hard consensus". But this practice makes it difficult to balance local and global information, and it neglects to negotiate amicably between local and global models to reach mutually agreeable results, called "soft consensus". In this paper, multiple-path solving method is proposed balance global and local features and combine these two feature preference paths to reach a soft consensus. Rather than relying on global information as the sole criterion, a negotiation process is employed to address the same objective by accommodating diverse feature preferences, thereby facilitating the discovery of a more plausible solution through multiple distinct pathways. Considering the overwhelming power of local features during local training, a swapping strategy applied to weaken them balance the solution paths. Moreover, to minimize the additional communication cost caused by the introduction multiple paths, the solution the task network converted into data adaptation to reduce the amount of parameter transmission. Extensive experiments are conducted to demonstrate the advantages the proposed method.
Introduction
Federated learning is a several training nodes to jointly train a unified global model without exchanging raw data 5, 6). able to federate data from multiple parties to capture a diverse data distribution 7 S As the significance of privacy has been increasingly emphasized [9 10 [1]. federated learning has been widely applied fields [12 with high privacy requirements, such as medical image processing [14 [15). recommendation systems [16] [7]. Internet of Things [18) and on.
distributed learning paradigm [ 2 3
that can coordinate
19 establishes an important training paradigm. Numerous subsequent works (20 21 are performed on the basis of this fundamental training paradigm. However, this training paradigm faces several challenges. Particularly, the problem of imbalanced data distribution 22 is widespread in the setting where federated learning is applied and can be attributed to the nature of distributed learning: joining multiple different data sources to participate in
As classic federated learning method, FedAvg
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the training process [23 24, 25 This would lead to disagreement among the optimization objectives
of the participating nodes [26 27).
The solution to the divergence of the optimization objectives at different nodes is usually to use global
information to constrain the local update process 28 29 + thereby enforcing the local optimization
objectives at nodes to be as close as possible to the global objective. This approach is referred to as hard consensus, as imposes rigid and immutable consensus across nodes. Typically, predefined indicators- Such as constraints on parameter updates or controls on the distribution of generated representations- are employed to regulate the optimization process. However, these indicators are often indirect with respect to the target tasks, as no direct criteria are available for task- evaluation. Furthermore, they are coercive nature, as the global norm remains fixed and unaltered. lacking flexibility or consultation. This rigid form of hard consensus may not always lead to optimal outcomes, as the global model may not perform better on local data than the existing client model.
50 1 The vanidhing of 2 The vaniohing of glubal fortanos
3E8
Cliea
50
local features
40
Cliere
Clbort P
10
Client
Client 10
Client
Cijerz
CliertE
Server
Approgaiaa
Misperg
Figure 1: The classification accuracies achieved by the model on the training set of each client are reported at three stages: initialization, after being trained by client A, and after aggregation. The global model, which is trained on local data, will improve the classification accuracy on the current client, while the knowledge delivered by the other clients will dissipate, this is reflected in the decreased accuracies on the training set of other clients. Similarly, aggregating the lo- cal models results in the vanishing of the local features. Global and local features are in an antagonistic relationship.
Aggraganen
anl snarping
Marping gloaa pesferanos
Stared projection
Thek natoorts
Figure 2: The architecture illustration of FedMP. On each client, the multiple paths solving is per- formed using mappings with different feature preferences and jointly optimizing the shared projection reach soft consensus between the global and local features. Clients only upload parameters other than the task network to save the communication costs. Multiple paths are balanced by an swapping strategy on the server.
Moreover, in federated learning, global and local features often compete with each other. A common approach to information exchange during each training round involves replacing the local model on the current node with the global model. However, this strategy can severely undermine the local features previously learned by the local model, leading to the dominance of global features. On the other hand, during the local training process, the local features have a greater advantage and take a dominant role, and continually influence the model under training. In this process, the dominance of global features would gradually vanish, the focus of the model turns to the local features, and the model would gradually forget what it has learned from the other clients. Global and local features alternately affect model training, these two features are in conflict, and are an unbalanced and oscillating state. However, they both play crucial role in the training of the final model. As shown in Figure I the local model has a high classification accuracy on the current client, however, it only receives its visible local data and therefore has very high local feature preference with the forgetting of the global knowledge, which leads to the lower classification accuracy on the other clients. The global model, on the other hand, fuses the knowledge of various clients, has global feature preferences, and therefore is able to achieve relatively high performance on all clients. Nevertheless, this also hits the local features, which can be seen from the performance decrease client A. To address this trade-off, it is essential to leverage both types of features and strike a balance between them, transforming their competitive relationship into a cooperative one.
For the hard consensus and the problem of local and global features balancing in federated learning, in this paper we propose multiple paths solving approach, called Federated Learning Framework
2

---

with Multiple Paths (FedMP), in order to balance the global and local features and prompt them to spontaneously generate a consensus, i.e., soft consensus. The architecture shown in Figure The proposed method integrates mappings with global and local feature preferences and federates their respective solution paths from distinct initialization points to achieve a consensus solution within unified solution space. The shared solution space projection, being directly related to the target task, serves as a task-driven criterion. This projection is learnable and is jointly updated alongside the multiple solution paths, replacing the previous indirect and static hard consensus mechanism. Additionally, a swapping strategy is introduced to further balance the influence of local features, fostering more harmonious cooperation between global and local features. Finally, the optimization objective is shifted from task-oriented to data-oriented, minimizing additional communication overhead in federated learning scenarios. Overall, the contributions of this paper are summarized as follows:
A soft-consensual federated learning framework is constructed via the multiple paths that effectively combine local and global information, providing platform for models with local features preferences and global features preferences to reach soft consensus.
The strengths of the paths balanced among each other, and the swapping strategy is used to weaken the overly strong paths.
To reduce the additional communication cost in federated learning scenarios, the solution target is shifted from the task to the data adapter for reducing the amount of parameter transmission.
The effectiveness of the proposed method is demonstrated using extensive experiments, and exploratory experiments are conducted for the multiple paths approach.
2 Methodology
First, we present a formal description of federated learning and the motivation for the proposed method in Section [2.] Then, in the following subsections, describe the detailed implementation of the proposed method.
2.1 Problem Description and Overall Motivation
For multiple device participating federated learning system, each participant is referred to client, and the p-th client has a dataset D, that is a subset of the plenary data D = (D1 U...U Dp), where P denotes the number of clients. The goal of federated learning is combine the data of all participants to generate a global optimal model. Specifically, due to the privacy issue, individual clients cannot exchange the raw data. Therefore, each client has its own optimization objective ming, C,(00,, Dp) under the model parameters Op. and the global objective is usually a weighted average of the local optimization objectives:
min C (0:D)
= Σ Co (0: (0:D,). p=1
P
Dpl
D
where |-| dedeoes the number of elements in the set and Ô means the global model parameters. During the training process, periodic information exchange is necessary to tune the training procedure of each client, preventing overfitting to the local data distribution and ensuring the model remains suitable for the global data distribution. The server, which acts the regulator of the federated learning system, typically broadcasts the current global model information at each communication round. This helps adjust the local models by providing a fresh optimization starting point. After the client completes the current training, the server accepts updates from the clients in order to collect the information provided by all participants, and aggregates them into the global model. The information delivered here is usually the parameters or the gradient of the model, and in some cases also contains other auxiliary information.
Motivation Due to the existence of non-lID data, global and local models typically exhibit conflict- ing feature preferences, and simply distributing or aggregating models can be detrimental to both global and local features. The global model, product of the aggregation from conflicting features,
3

---

contains information that may not all be absolutely correct. Therefore, the mechanism of using global information as a hard criterion to drive local models in order to create hard consensus between local features and glohal features has potential for improvement. Moreover, using global information as reference often lacks consideration of the target task, and a task-driven coordination approach is worth investigating. How to harmonize global and local information, preserve as complete information as possible with balance among them, and make them produce mutually agreeable results on the target task to reach soft consensus is the focus of this work.
2.2 Multiple Paths High Confidence Solving with Information Relics
common approach to information transfer federated learning is model transmission. Through the use of global model sending and replacing the local models, so as to use client specific data to adjust the mean of the information provided by the previous clients involved in the training. And global information may also be used as the criterion to control local updates. This is a very intuitive way, but lacks consideration of the balance between global and local information, and the information relics of local features. We believe that a softer approach with more comprehensive reference information should be used, and reach a soft consensus on global and local models. Therefore, the proposed multiple paths solving approach is described below.
For typical neural network, there is stream of data directed from the inputs to the outputs, which maps the input space to the output space. We have the following definition:
Definition 1. (Solution path). It is expected that there is a mapping f((:) which makes the data x
map to the target y = f (z). Fitting this mapping using a neural network g()), defines the process of optimizing g(r) to (x) as the solution path. Here, l1g* (x) - f(x)]l S e, and is the error.
Definition 2. (Different solution paths), For mapping f-), the optimization of neural network gi (-) g (-) and neural network 92(-) to 9-(-) will be the different solution paths. Here, (197(() f(x))] S
(1 (1))(x)) - f(x)] S 62, where 61 and 62 are the errors.
From the above definitions, it follows that the model trained locally encapsulates knowledge refined from the data of single client, resulting in one solution path, while the global model distributed by the server, which incorporates knowledge refined from the data of all participants, leads another solution path. Relying on single solution path often results in biased information and increases the likelihood of the neural network becoming trapped in local optima. To address this issue, information from multiple solution paths should be utilized simultaneously. This approach ensures the aggregation of diverse information sources, preventing the overwriting and forgetting of knowledge associated with single information channel. Furthermore, the concurrent use of multiple solution paths offers an alternative to the traditional practice of using global information as a rigid criterion, promoting a more flexible and adaptive learning framework. For the mapping from input to output on client p. constructed using the multiple paths way, it can be trained by the following loss function:
iDpl
Cp
1
1 (V
( (X,))) C60 (V (V (X,)))
PDpl
where X- E Dr is the i-th data sample of the p-th client, c/ and clS) denote the loss functions for the local and global solution paths, respectively, y is the path balancing hyperparameter, 5000 (-) and 0 (.) represent the mappings under two different solution paths, which map the input data to the latent space, and 1 (-(()) denotes the projection that maps the data in the latent space the eventual output space. Note that here (-) is shared by all solution paths. The shared projection serves to make the objectives of all multiple solution paths located in the same solution space, and to ensure that all solution paths have the same target by employing the same loss function. Solving for an identical objective from different paths leads solution with very high confidence. This based on a highly intuitive assumption: if different approaches yield the same or very similar results, then such results can be considered highly reliable. Therefore, by using this loss function, the mappings are solved from both solution path biased towards local features and a solution path biased towards the global features, which gently collaborates the global and local information, and prevents overwriting and forgetting of the information. The soft consensus generated by the global and local information is materialized in shared projection, and the mutually agreeable results are reached through their respective solution paths. The shared projection guarantees that the space of solutions is mapped in

---

way that is more suitable for reaching consensus and is directly related to the label space, which is task-driven. Further discussion on multiple paths can be found in the Supplementary Material.
2.3 Local Preference Swapping Strategy
From the multiple paths solving process described in Section 2.2 the following sequences of initial models in each communication round are given the p-th client:
L p { pt
2/2
R J
(3)
(1) = 10 p:
(4)
$ ((s) p {Ør
//"() (-) r-th communication round p- where of denotes the model used for approximate mapping 4 th client, and v/p is the model used for approximate the projection V V,).). Here, d' -Σ [2 of
When r = O, the whole system is in initialization state, so that 0/ = 09.
In federated learning, restriction on access to data is a key factor affecting model learning. Local models trained too much on the current dataset would overfit the visible distribution and have difficulty in adapting to the global distribution even if there is the multiple paths method to correct for it. Therefore, the multiple paths network for federated learning needs to be modified in terms of training strategy.
easy find that all conditions required for the multiple paths method can be summarized as, a solution path with a preference for local features and a solution path with preference for global features, and to optimize on both solution paths simultaneously. Therefore, it only necessary to provide solution paths under other mapping preferences which are different from the global features to satisfy the requirements for the multiple paths formulation. A local preference swapping strategy can be used to avoid the local model overfitting problem by replacing the local model in the current client with random local model of other clients downloaded from the server at the beginning each communication round. For the p-th client, we can convert the sequence of initial models Eq. (4) and Eq. (3) to the following form:
(1) p
10p: o
R D
(6)
= (w) 122 p2r,
DR }.
(7)
where p" a random sample the client indexes that participated in the previous round of training. In this way, each local model of the participating clients is updated after each round of communication, so that the multiple paths method obtains the optimized origin of the solution path that is preferred to the local features, and avoids the overfitting because the models do not stay one client all the time. Additionally, the swapping strategy enables each model to accept a wider range of inputs, which improves the generalization ability of the model.
2.4 Multiple Paths Data Adaptation
In federated learning, communication cost is topic worth considering. Compared to the classical federated learning methods such as FedAvg, multiple paths solving would require additional network communication cost. Specifically, at each communication round, the contents to be exchanged include the mapping with local feature preferences and the mapping with global feature preferences. Although the additional communication cost of multiple paths solving is not too large compared to methods such as SCAFFOLD that require uploading large amount of auxiliary information, we still consider further reducing the additional communication cost associated with multiple paths solving.
Originally. we employed the multiple paths solving directly to the full task network, which would have transmitted the network parameters under different multiple solution paths. The communications volume is positively correlated with the number of current task network parameters. A mind shift can be made to shift the target of multiple paths solving from the task network to the adaptation of the data, and to relax the optimization target for the full task network, relocating the more stringent optimization target to the adaptation of the data. The scale of the data adaptation network can be small, but the solution objective is precise, and the exchange of information among the individual
5

---

clients only occurs over this part of the network parameters, which greatly reduces the communication cost during each communication round. Therefore, a modification of Eq. function:
2 yields the following loss
p
hp
( ( (2p((T)))))))) (T))) +76, (V (V,(9)(T)) (T()))))))))
D
(8)
s.t. 11) = 1 ) p [())+
1 (x)).
( Here, (2p()) denotes the full task network the p-th client, Tp and (g),i represent the adapted data generated from the two mappings on different solution paths, respectively. (2p(.) remains constant during the multiple paths solving process, in which the fitting of the network to the task is transformed into the fitting of the data to the network, i.e., the data attempts with some rules to mimic the patterns that can be recognized by the network. This is shift in the goal of the solution and this shift needs to be solved by multiple paths solving in order to obtain a more accurate solution. For the task network, extracting the hidden deep information in the data requires very complex network with huge amount of network parameters to be updated to suit the task. While by simply adapting the data to convert them into the features which are required for the task network, the size of the network required can be tiny since this does not require the decomposition of the deep information in the data. This may intuitively diminish some of the performance, but we are surprised to find that conjunction with the multiple paths solving method, the expected performance degradation is not very noticeable, which can be verified by the subsequent experimental results.
2.5 Training Details
Since the multiple paths solving method requires different solution paths for the solving process, and we want all the solution paths employed are reliable. Hence, in the early stage of training, we use few global communication rounds employing the same training strategy as FedAvg in order to obtain more reliable solution path with global feature preferences. At this stage, the model p-th client is trained using the following loss function:
IPol Dpl Σ (2,((((),, s.t. - (x)))
Cp
And, after each communication stage, the same global model is used to replace the client model for each client. After completing the initial training process, the parameters of the task network (.) on p-th client are not changed and the parameters of these networks are no longer exchanged in the communication. Then, for the p-th client, the multiple paths solving process performed using Eq. (8). and the local preference swapping strategy proposed in Section 2.3 used to allow each client to obtain a more reliable solving path with local feature preferences. Additionally, considering that the performing of the swapping strategy after several communication rounds may result in the mappings preferring local features overly preferring local features, we use the mappings with the global feature preferences every B rounds for replacement in order to balance the influence of the local features. Since the models are only exposed to local data during training, only the mappings with preferences for local features require additional balancing. The detailed algorithm is summarized in Algorithm (See Supplementary Material).
3 Theoretical Analysis
In this section, we provide a convergence analysis of the proposed FedMP. First, some assumptions are introduced to help complete the following theoretical analysis.
Assumption 1. For any E [P), local loss function for local solution path ( and local loss function for global solution path cla) are L-smooth with respect O For V O and O' the following inequalities hold:
I IV (0) (C'I 1 -eIl,
(10)
| (0) (C') < T2||0 e'll
(11)
where L1 and Le are Lipschitz constants.

---

Assumption 2. The upper bound on the variances of the local gradient to the aggregated mean can be given as follows
- (0) = 6%.
(12)
P
(V) VZ)
(0)1 م &
(13)
p=1
where o, and ōg are the constants.
Thus, Theorem can be derived:
Theorem 1. Suppose Assumption Hand hold, the convergence property of the proposed method can be described by
E [IV s.
(14)
R
where = - )
+ S, and
7.1l
H = BE
FT, + 712) F2 + )12)
2
(15)
-61 B² (I, + 712))
EBB
S
+ 2 22 (L, + )22
B4 + * + K
(16)
p (6) + 22)))
From Theorem Theorem we can know that the proposed FedMP can reach the convergence with appropriate hyperparameters choices. See Algorithm for definitions of R, E, B, and K.
Proof. Please see Supplementary Material for detailed proof.
Experiments
4.1 Experimental Setup
Datasets: We conduct main experiments with the proposed method using classification tasks on three datasets, which are CIFAR-10 10 (30), CIFAR-100 30 and Flowers (31. CIFAR-10 and CIFAR- have 10 and 100 classes, respectively, and Flowers 102 has 102 classes. For CIFAR-10 10 and CIFAR-100 datasets, we use a train set consisting of 50,000 samples and test set consisting of 10,000 samples. For Flowers 102 dataset, we use a train set consisting of 6, 1.4 samples and a test set consisting of 1,020 samples. We partition the train sets using the Dirichlet distribution with hyperparameter a E {0.3, 0.5, 1.0} to simulate the scenarios with the heterogeneous data distribution, where the smaller is, the more unbalanced the data distribution among clients. The distributions of the train set and other settings can be found Supplementary Material.
Baselines: In order to compare the results, we also run nine SOTA federated learning methods under the same datasets setup, including FedAvg [19, FedProx (20). SCAFFOLD 21], FedNova [32]
FedDF 33,, MOON (28), FedASAM [34), FedPVR (29] and FedUCS 12]
Experimental results are shown in the following and the Supplementary Material.
4.2 Experimental Results
4.2.1 Experiment of Multiple Paths
We first validate the effectiveness of multiple path solving centralized learning. We use the three test datasets the training data for the classification task. The MLP module is used as the shared
7

---

SRdi JEX Pill
83
Los Ruadinn Vikae
Runton Vilae
-Le Sace
Los Baskin 12
aac
ACC
20
Epects
Tpocts
Spocts
(a) CIFAR-10
(b) CIFAR-100
(c) Flowers102
The curves of loss function values and classification accuracies during training for single path solving and multiple paths solving. Multiple paths solving can make the model find a fitter solution faster and get better performance.
Figure 3:
projection and two ResNet-18 modules are used as the multiple solution paths. Since the definition of global and local features does not exist in centralized learning, we use two different initialization methods two different solution paths. One of the solution paths is taken out individually and used the same training setup for the task network under the single path, and this network is also used as the task network in the multiple path setup as well. The loss and training accuracies curves that vary with epochs during training are demonstrated in |Figure 3) From the figure, it can be seen that the task network in multiple paths can obtain faster learning speed and better final learning results than the single-patt network without changing the network structure and the optimization method settings. The multiple paths solving method can combine the multiple solution paths to form a soft consensus, and avoiding the limitations of single path solving. The knowledge learned by different paths can effectively provide assistance among paths, and prevent single path to be trapped in a local optimal solution. Therefore, we can think that the multiple path solving method can effectively help the network obtain more comprehensive and reliable information and improve the training performance by reaching consensus among multiple paths. The experimental results demonstrate that the proposed multiple paths solving method effective. And this effectiveness is the strong motivation for us apply this method in federated learning.
4.2.2 Main Results
Here, we show the classification accuracy results of the proposed method with eight other comparative methods three datasets in Table It can be seen that the proposed method is able to better optimal performance in most of the dataset settings. The proposed method is able to achieve greater advantage under the scenarios of higher data heterogeneity or the more difficult classification tasks, which we attribute to the fact that the multiple paths solving approach provides more comprehensive auxiliary information and replaces the hard consensus by forming soft consensus between global and local features, and reversing the information that may be biased in the global information. With the multiple paths solving, the global and local information, which are opposing to each other, are converted into cooperative relationship in order to perform federated learning more comprehensively from different perspectives.
Table 1: The results of three datasets for all methods. Bold and underlined results are the best and the second best. "1" indicates accuracy improvement over FedAvg.
CIFAR-10
CIFAR-100
Flweer
a=0.3
a=0.5
a=1.0
a=0.3
c=0.5
a=1.0
a=0.3
a=0.5
a=1.0
FedAvg
80.66
83.34
87,46
57.25
$9.13
61.34
41,86
43.92
49
FedPhox
81.16
83.67
88.04
56.37
$7.77
62.36
34,90
37.35
39.22
SCAFFOLD
80.60
86.08
87.66
58.65
60.96
61.81
45.98
46.37
48.92
FedNova
83.41
84.93
86.41
55.46
55.02
56.67
41,08
40.69
43.73
FedDF
78.70
76.95
38 18.651s.19
22.96
24.09
MOON
82.47
85.46
87 83
57.52
59.34
62.85
41,08
43.33
45.88
FedASAM
71.59
74.14
30.59
FedPVR
$4.65
85.73
87.86
59.32
60.89
62.75
43.04
44.41
45.88
FodUCS
47.79
89.43
62.19
63.24..11
64.28
43.72
19.22
51.18.n.0
FeaMP
85.37
88.96
88.31
61.414.38
62.04 12.9
62.95
43.73
11.41
13.43
4.2.3 Ablation Study
In order to validate whether the various proposed enhancement strategies are effective in the federated learning scenario applied to heterogeneous data, perform the ablation study and demonstrate
8

---

the results in Table 2 Specifically, for the two main strategies we proposed: multiple paths solving and swapping strategy, we make them available or not and train the frameworks constructed by all possible configurations under the same dataset setup, yielding the results in Table 2] Taking the first row of data in the table as the baseline, from the other data in the table we can see that when using only the multiple paths approach, since one of the paths uses the local models without swapping, the model predictably fits more strongly to the local data, and leading to the overfitting for the local features. Therefore, this path will affect the learning of global features, and even with the assistance of global features, it is still unable to form a more correct consensus, so that the performance decreases compared to the baseline. The model that only employs the swapping strategy is unable to obtain the update of global features in time, and thus cannot obtain a stable improvement for the learning of the global model, which is rather negative impact on the global model at the CIFAR-10 dataset with the heterogeneous parameters of 0.3 and 0.5. In other settings, there are boosts, but the magnitudes more limited. In contrast, when both proposed strategies are applied to the model, the more significant and stable boosts are obtained in most settings.
In order to demonstrate more intuitively the ef- fectiveness of the proposed multiple paths solv- ing approach in federated system, the results of the classification accuracy achieved on the training data of each participating client for both local models and global models are presented in infiuuee 4] From the figure, it can be seen that when using the multiple paths solving approach for training, it is usually possible to achieve more excellent performance, both for the local and global models. When using a single-path architecture, the absence of a global path elim- inates the need to account for global features, allowing the model to better fit local data. How- ever, this also limits the single-patl approach's 's ability to generalize to data from other clients. Additional experimental results are provided in the supplementary material.
ACCOS
Cirrt
20
BMoilyyle parks @Siehepeh
iMMaigle pats @SSigle
(a) Local model of Client
(b) Global model
Figure 4: Classification accuracies of the two mod- els on the data in different clients. The local model trained by the multiple paths able achieve bet- ter performance on the data of other clients, and it can increase the performance of the aggregated global model.
Table Ablation study of FedMP on three datasets. Bold results are the best.
CIFAR-10 a=D.5
CIFAR-100
Flowers 102
Multiple Paths
Swapping Strategy
a=0.3
a=1.0
a=0.3
a=0.5
a=1.0
o=033
a=0.5
a=1.0
83,66
85.32
85.45
60.40
61.06
62.29
38.33
52.32
49,80
83.16
84.97
85.50
60.56
61.31
62.79
43.04
52.12
54.12
82.96
81.47
82.82
56.97
58.26
59.71
38.63
51.86
49.41
85.37
88.96
88.31
61.43
62.04
62.95
43.73
54.41
53.43
4.3 FedMP with Full Multiple Paths
Here, instead of using the communication cost reduction method proposed in Section 2.4, we use multiple paths solving method for the complete task network. We use the complete task network here as the mapping with global feature preferences and the mapping with local feature preferences, respectively, and use the same shared projection as in the experiments to motivate a soft consensus. We show the classification accuracy results in Figure 5) It can be seen that the results obtained are further improved after using the full multiple paths (FedMP-F). However, the penalty for this enhancement would be a higher communication cost, and we believe that the small amount of performance degradation associated with the reduction in communication cost is acceptable. Therefore, there is flexibility to trade-off between the two approaches depending on the requirements.
4.4
Data Privacy
As
machine learning paradigm, federated learning avoids the transmission of raw data, thereby providing a certain level of privacy protection for participants. However, it remains possible to infer sensitive information about the original data from model parameters updates. To evaluate the adaptability of additional privacy-enhancing strategies, we conduct further experiments. Differential Privacy (DP), well-established protection mechanism, is employed by injecting noise
9

---

into the transmitted model parameter updates. We use a backbone network consisting of two CNN layers and two fully connected layers, and apply DP with noise multiplier of 1.0 on the CIFAR-10 dataset. The results are shown in theuFi 6 As the table indicates, even with the addition of privacy- preserving mechanisms, the proposed method still maintains relative performance advantage. This demonstrates that FedMP can be extended with such strategies to meet stronger privacy protection requirements.
90
PedMP
FecAvR
PezAvp-DP
FelMP
Accitiy (50
40
2 655
FeiMI-DP
70
64
60-
e1.43
52.99
62
K2.8
+1.09
62.19
6115
E107
50
0.5
0.5
1.0
0.3
0.5
1.0
60 0.5
CIEAR-10
CIFAR:100
0.5
1.0
Dascect
CIFAR-16
Figure 5: comparison of global model per- formance with and without the complete multi- path network. The results show that adopting the full network leads to improved performance of the global model.
Figure Comparison of global model perfor- mance with and without differential privacy un- der different data settings. The results show that FedMP maintains its advantage even when privacy- preneerie mechanisms are applied.
4.5 Parameter Sensitivity Analysis
This subsection explore the impact of path balancing parameter y on the multiple paths solution effect. On the CIFAR- 100 dataset, by setting different path balancing parameters values and recording the test accuracy. the results are shown in Table 3] As shown the table, when the path balaneing parameter is appropriately tuned, the muluple paths solving method achieves relatively stable and excellent performance. However, if the parameter is set too high or too low, it adversely impacts the final performance. This negative effect is particularly pronounced when the global path dominates. We speculate that this is because the global path is derived from the aggregation of local solution paths. Assigning excessive weight to the global path disrupts the optimization of local paths. In turn, poorly optimized local paths negatively influence the aggregated global path, amplifying the adverse effects in subsequent multiple paths updates. Thus, balancing the two paths is critical. An appropriately balancing parameter enhances the effectiveness of the multiple paths solving method, allowing it to achieve better performance.
Table The results of classification accuracy under different path balance parameter values on the Bold and underlined results are the best and the second best.
CIFAR- 100 dataset.
7
0.01
0.05
0.1
0.5
100
0=0.3
60.76
60.46
61.43
60.17
59.22
2.57
23.92
1.00
1.00
0=5.5
61.35
61.13
62.04
60.91
60.18
9.19
16.33
1.00
1.00
a= 1.0
63.10
62.94
62.95
62.89
62.76
19.21
14.50
1.00
1.00
5
Conclusion
In this paper, we propose a federated learning framework with multiple paths for balancing global and local features, and motivating them to reach a soft consensus. By introducing multiple paths solving method, mappings with global and local feature preferences are federated to provide multiple solution paths for reaching soft consensus among them. Since the balance of multiple solution paths is important, additional swapping strategy is introduced to equalize the strength of paths. Switching the objective of solving from the task network to the adaptation of data reduces the communication cost. We have performed various experiments, including exploring the effectiveness of multiple paths, to demonstrate that the proposed federated learning framework with multiple paths effective. the future, we will explore better ways of constructing multiple paths to achieve better performance.
10

---

Contribution Statement
Sheng Huang and Lele Fu contributed equally to this work.
Acknowledgments
The research is supported by the National Key R&D Program of China (2023YF82703700) the National Natural Science Foundation of China (62176269).
References
[1] Xiaoli Tang and Han Yu. Reputation-awai revenue allocation for auction suction-based federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 39, pages 20832-20844 2025.
[2] Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogeneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56(3):1-44, 2023.
[3] Ming Hu, Yue Cao, Anran Li, Zhiming Li, Chengwei Liu, Tianlin Li, Mingsong Chen, and Yang Liu. Fedmut: Generalized federated learning via stochastic mutation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12528 12528-12537, 2024.
[4] Tianchi Liao, Lele Fu, Jialong Chen, Zhen Wang. Zibin Zheng, and Chuan Chen. A swiss army knife for heterogencous federated learning: Flexible coupling via trace norm. In Advances in Neural Information Processing Systems, volume 37, pages 139886-13991 2024.
[5] Lele Fu, Sheng Huang, Yanyi Lai, Tianchi Liao, Chuanfu Zhang, and Chuan Chen. Be- yond federated prototype learning: Learnable semantic anchors with hyperspherical contrast for domain-skewed data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 16648-16656, 2025.
[6] Lele Fu, Sheng Huang, Yuecheng Li, Chuan Chen, Chuanfu Zhang, and Zibin Zheng, Learn the global prompt in the low-rank tensor space for heterogeneous federated learning. Neural Networks, 187:107319, 2025.
[7] Jinyu Cai, Yunhe Zhang, Jicong Fan, and See-Kiong Ng. Lg-fgad: An effective federated graph anomaly detection framework. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 3760-3769, 2024.
[8] Mingzhe Chen, Zhaohui Yang. Walid Saad, Changehuan Yin, H. Vincent Poor, and Shuguang Cui. A joint learning and communications framework for federated learning over wireless networks. IEEE Transactions on Wireless Communications, (0(1):269-283 2021.
[9] Xiaoli Tang and Han Yu. A cost-aware utility-maximizinn bidding strategy for auction-based federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[10] Xiuwen Fang, Mang Ye, and Du. Robust asymmetric heterogeneous federated learning with corrupted clients. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[11] Xiuwen Fang and Mang Ye. Noise-robust federated learning with model heterogeneous clients. IEEE Transactions on Mobile Computing, 2024.
[12] Sheng Huang, Lele Fu, Yuecheng Li, Chuan Chen, Zibin Zheng, and Hong-Ning Dai. A cross-client coordinator in federated learning framework for conquering heterogeneity. JEEE Transactions on Neurul Networks and Learning Systems, (8(5):8828-882, 2025.
[13] Zhuang Qi. Lei Meng, Zhaochuan Li, Han Hu, and Xiangxu Meng. Cross-silo feature space alignment for federated learning on clients with imbalanced data. In Proceedings of the AAAI Conference Artificial Intelligence, pages 19986-19994, 2025.
[14] Qian Yang, Jianyi Zhang, Weituo Hao, Gregory P Spell, and Lawrence Carin. Flop: Federated learning on medical datasets using partial networks. In Proceedings of the ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 3845-3853, 2021.
11

---

[15] Meirui Jiang, Zirui Wang, and Qi Dou. Harmofl: Harmonizing local and global drifts in federated learning on heterogeneous medical images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1087-1095, 2022.
[16] Ben Tan, Bo Liu, Vincent Zheng, and Qiang Yang. A federated recommender system for online services. In Proceedings of the ACM Conference on Recommender Systems, pages 579-581, 2020.
[17] Khalil Muhammad, Qinqin Wang, Diarmuid O' O'Reilly-Morgan, Elias Tragos, Barry Smyth, Neil Hurley, James Geraci, and Aonghus Lawlor. Fedfast: Going beyond average for faster training of federated recommender systems. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1234-1242, 2020.
[18] Xiaoli Li, Shixuan Li, Yuzheng Li, Yuren Zhou, Chuan Chen, and Zibin Zheng. A personal- ized federated tensor factorization framework for distributed iot services qos prediction from heterogeneous data. IEEE Internet of Things Journal, 9(24):25460-25473, 2022.
[19] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication
learning of deep networks from decentralized data. In Proceedings of the International Conference on Artificial intelligence and statistics, pages 1273-1282, 2017.
[20] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Proceedings of the Machine learning and Systems Conference, pages 429-450, 2020.
[21] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the International Conference on Machine Learning, pages 5133,,,4,, 2020.
[22] Zhiqiang Kou, Jing Wang. Jiawei Tang, Yuheng Jia, Boyu Shi, and Xin Geng. Exploiting multi-label correlation in label distribution learning. In Kate Larson, editor, Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 4326-4334. International Joint Conferences on Artificial Intelligence Organization, 8 2024. Main Track.
[23] Yichen Li, Yijing Shan, Yi Liu, Haozhao Wang, Wei Wang, Yi Wang, and Ruixuan Li. Per- sonalized federated recommendation for cold-start users via adaptive knowledge fusion. In Proceedings of the ACM on Web Conference 2025, pages 2700-2709, 2025.
[24] Yichen Li, Haozhao Wang, Yining Qi, Wei Liu, and Ruixuan Li. Re-fed+: A better replay strategy for federated incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[25] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Kindle federated generalization with domain specialized and invariant knowledge. IEEE Transactions on Information Forensics and Security, 20:3461-3474, 2025.
[26] Farshid Varno, Marzie Saghayi, Laya Rafiee Sevyeri, Sharut Gupta, Stan Matwin, and Moham- mad Havaci. Adabest: Minimizing client drift in federated learning via adaptive bias estimation. In European Conference on Computer Vision, pages 710-726, 2022.
[27] Divyansh Jhunjhunwala, Pranay Sharma, Aushim Nagarkatti, and Gauri Joshi. Fedvarp: Tack- ling the variance due to partial client participation in federated learning. In Uncertainty in Artificial Intelligence, pages 906-916, 2022.
[28] Qinbin Li, Bingsheng He, and Dawn Song. conta-cone federated learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10713-10722, 2021.
[29] Bo Li, Mikkel N. Schmidt, Tommy S. Alstrøm, and Sehastian U. Stich. On the effectiveness of partial variance reduction in federated learning with heterogeneous data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3964-3973, 2023.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

[31] Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, 2008
[32] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor Tackling the objective inconsistency problem in heterogeneous federated optimization. In Proceedings of the Conference and Workshop on Neural Information Processing Systems, pages 7611 7611-7623, 2020.
[33] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems, pages
2020.
[34] Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In Proceedings of the European Conference on Computer Vision, pages 654 672, 2022.
[35] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10174-10183, 2022.
[36] Jian-hui Duan, Wenzhong Li, Derun Zou, Ruichen Li, and Sanglu Lu. Federated learning with data-agnostic distribution fusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 8074-8083. 2023.
[37] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16312-16322 2023.
[38] Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8397-8406 2022.
[39] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heteroge- neous federated learning. In International conference on machine learning, pages 12878 12889, 2021.
[40] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Fedde: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 10112-10121, 2022.
[41] Jie Zhang, Zhiqi Li. Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In Proceedings of the International Conference Machine Learning, pages 26311 16329, 2022.
[42] Yutong Dai, Zeyuan Chen, Junnan Li, Shelby Heinecke, Lichao Sun, and Ran Xu. Tackling data heterogeneity in federated learning with class prototypes. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7314-7322. 2023.
[43] Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se-Young Yun. Preservation the global knowledge by not-true distillation in federated learning, Advances in Neural Information Processing Systems, pages 18461-38474, 2022.
[44] Jiawei Shao, Yuchang Sun, Songze Li, and Jun Zhang. Dres-fi: Dropout-resilient secure federated learning for non-iid clients via secret data sharing. Advances in Neural Information Processing Systems, pages 10533-10545, 2022.
[45] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263. 2021.
13

---

[46] Rui Yan, Liangqiong Qu, Qingyue Wei, Shih ChhnnCheng Huang, Liyue Shen, Daniel L. Rubin, Lei Xing, and Yuyin Zhou. Label-efficient self supervvee federated learning for tackling data heterogeneity in medical imaging. IEEE Transactions Medical Imaging, 42(7):1932-1943, 2023.
[47] Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference Machine Learning, pages 18250-18280, 2022.
[48] Grigory Malinovsky, Kai Yi, and Peter Richtárik. Variance reduced proxskip: Algorithm, theory and application to federated learning. Advances in Neural Information Processing Systems, pages 15176-15189 2022.
[49] Yuecheng Li, Tong Wang, Chuan Chen, Jian Lou, Bin Chen, Lei Yang. and Zibin Zheng. Clients collaborate: Flexible differentially private federated learning with guaranteed improvement of utility-privace trade-off. Proceedings of the Forty-second International Conference on Machine Learning, 2025.
[50] Michael Kamp, Jonas Fischer, and Jilles Vreeken. Federated learning from small datasets. The Eleventh International Conference on Learning Representations, 2023.
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[52] Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 2528-2535, 2010.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The related contents appear in the abstract and the final paragraphs of Intro- duction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification:
We discuss the potential risks of the proposed method in Section B.2 of the
Supplementary Material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification:
The complete proof is provided in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental setups are described in Section 4.J of the main text and Section E. of the Supplementary Material, with detailed algorithmic procedures provided in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We plan to release the code, datasets, and pre-trained models with sufficient instructions to faithfully reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cor
public/guides// wow.iideee/oeSSiiiiiiiiiaeaa www.hbmmmssioooooiocicco for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips wwtpccpppccc..cc http:pulliccceeee wwtcc//gggs////eeee Codeeeumissiiiiiicc.ce.a.. www.ssiiiciiccccycc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Experimental setups are described in Section 4. of the main text and Section E. of the Supplementary Material, with detailed algorithmic procedures provided in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our experiments do include significance experiments and therefore do take into account statistical error information.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resourees information in Section E.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //nnuurrps.... Setublic/Ethicsaaiiee ines?
Answer: [Yes]
Justification:
The research is consistent in all respects with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the significance of algorithms for the real world in the Introduction section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Inapplicable
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All codes and datasets used in this work are publicly available for research purposes.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, patasets.paperssee paperswi theode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Inapplicable
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Inapplicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Inapplicable
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //nerr//euurippp /ttprp/cccc///eeeeeeee///////// for what should or should be described.
21

---

A Related Work
Federated learning needs to be performed without exchanging raw data in order to protect the privacy of all participants [35] 36 37.. This paper focuses on federated learning in non-llI scenarios, which is the case where the distribution of data is heterogeneous distributed clients. As pioneering work on federated learning, FedAvg [99 joins distributed data for training by communicating model updates to achieve the target that several nodes co-train the global model. However, it leads to performance degradation in scenarios where data distributions have a large variance, which is used to process
non-IID data [8 39 There a number of works 40
44 that have been proposed
to tackle the impact of data heterogeneity on federated learning. SCAFFOLD [21] introduces a method for controlling local updates by utilizing variance reduction techniques on client-side updates to overcome the effects of client drift. Similar idcas for controlling client-side updates have been adopted by various works, e.g., FedProx (20) controls the divergence between the local and global models by proposing proximal regularization term to minimize the difference of the local model
and the global model, FedDyn 44 employs a similar technique. MOON [28] makes each local update process not to deviate too far from the global model through contrastive learning. FedPVR [29] corrects for client updates by variance control at the end of the local models. Above and other works 46 47 48 control the direction of client-side optimization by enshrining global information as the norm, which produces a hard consensus that limits the preservation of local features that may be beneficial to the final model. And, there is also lack of consideration of the balance between the global and local features.
B
Discussions
B.1 More Discussion of Multiple Path
Under the concept of solution path, an intuitive illustration in Figure 7 shows the difference be- tween the proposed multiple paths method and other ways of training neural networks. Many federated learning methods use the following approaches in the process of local updating. For
Mappin
Guiding
Solatice poth
Sulutiion path
Guidina
Loss
Marpins
Loss
Marriti
a classical neural network as shown in Figure 7ag it is necessary to accept an input and make the
(a) Classical neural network (b) Contrastive learning
neural network generate an output that is as close
Sollation path
Guiling
Sclaice
as possible to the expected value with the guid- ance of the loss function. There is only one solution path in this training approach, which is to adjust the parameters of the network to a more optimal value based on the current inputs and outputs of the network. For the network with contrastive learning as shown in Figure 7b] group of inputs need to be accepted and positive and negative pairs need to be defined in order to bring the representations of the positive pairs closer and pull the representations of the negative pairs farther away. The solution path in this approach can be considered the guidance for optimizing the network by data similarity, and there is still only one solution path. For the knowledge distillation as shown in Figure 7c| it is required and a teacher network, that receive the same inputs and use the output of the teacher network as a criterion for tuning the parameters of the student network. This approach employs the information provided by the teacher network as the additional knowledge for the solution path of the network parameters optimizing, which is similar to contrastive learning, also with only one solution path, although it looks like two. The proposed multiple paths solving as shown in Figure 7d]is similar to the knowledge distillation method in that two different networks receive the same inputs, however, the difference is that the two networks with different paths are solving for the same objective at the same time. By sharing the projection (-) and the same loss function, the two networks are ensured to be in unified solution space in the latent space, which thus ensures that they have the same solution target. The multiple solution paths can be different mappings or different mapping preferences. This approach can be described as All roads lead to Rome. We use different mapping preferences two
Ma Maypiing
Macening
(c) Knowledge distillation
(d) Multiple path solving
Figure 7: An illustration of the difference between the proposed multiple paths method and other ways of training neural networks.
use a pair of networks, which includes student network
22

---

different solution paths, because there are two completely different feature tendencies in federated learning, i.e., global features and local features. Therefore, multiple paths solving very suitable for federated learning.
B.2 Discussion about the Swapping Strategy
In this paper, propose multiple paths solving approach for federated learning, and swapping strategy is used in order to obtain more reliable solution paths. This subsection discusses the performance risk and privacy risk of the swapping strategy.
Performance risk The swapping strategy requires additional model parameters processing and introduces additional communication costs. This increases the computational and communication consumption of the federated learning system. Therefore, to mitigate this performance risk, we propose multiple paths data adaptation mechanism the paper. This mechanism converts the solving of the task network into the adaptation of data, reducing the number of model parameters that need to be exchanged.
Privacy risk The risk privacy leakage that may result from the swapping strategy is explained here. Other clients would receive model of some client from the server when the swap occurs, which may raise the privacy leakage of the original training data. Moreover, during the transmission of the models, there is also the possibility of information leakage due to unreliable communication links. But, there are still possible solutions to the above privacy risks. Since the execution party of the swapping strategy is the server, for the models that are distributed, the each client receives model that is usually from a random client, and the model has been trained by several clients before is distributed to the current client, which make it difficult to infer the gradient and personalization information of a specific client. Hence, increases the difficulty of the privacy attack. In addition, each round of model assignment is randomized. which leads to difficulty in tracking the flow of specific models, which further protects client privacy. By applying differential privacy technique 49 50 to the transmitted data, the protection of privacy can be obtained even further.
C Algorithm Details
We present the complete algorithmic workflow of the proposed FedMP in Algorithm| [ provides the detailed client-side optimization workflow implemented in Algorithm And, summarize all symbols used in the paper in Table 4.
Algorithm
Symbol
Meaning
D
The dataset of the p-th client, which is subset of the global data The number of elements in set
I|
0
Global model parameters Model parameters the y-th client The i-th data sample of the y-th client
0
X
C
The loss function for the local solution path on the p-th client
The loss function the global solution path on the p-th client The path balancing hyperparameter, used to balance the influence between local global paths The mapping of the local solution path the p-th client
7
6.9 (+)
The mapping the glohal solution path on the p-th client
The shared projection on the p-th client
The model used by the p-th client in the r-th communication round to approximate 4,((.) The model used the p-th client to approximate the projection V.(.)
p
A random sample client indices that participated the previous round training
2.(1)
The task network on the p-th client
The adapted data generated on the p-th client of the i-th data sample through the local path mapping The adapted data generated on the p-th client of the i-th data sample through the global path mapping
Table 4: List of symbols and their meanings.
23

---

Algorithm Federated Learning Framework with Multiple Paths (FedMP)
Input: The number of communication round R, The number of initial training round Ric. The number of balance round B, the number of local epochs E, hyperparameter the learning rate r) and the number of selected clients per round.
Output: The global model + $2
Server:
2: Initialize the weights v/ w and 6 of v, l2 and Φ, respectively;
3: for 11 R do
4:
C C (Randommly selects K clients); for p-th client E C
5:
6:
if Rst then
7:
1/ 08 f Client(p, r, "",,"""-,,,,,, ,,0,,,0000 ,0
8:
else
10 [0p] + out que_qqueeMM"
9:
if R.% 0 then
11:
10 ppp { :1=11;;
12:
end
13:
Wp or Client(p, Vr. @11) ,
15:
end if
end for
17:
Press UpEc } into queue M
18:
[""; }0} + Epec Epec D2I (v] op):
pPpl
end for
20:
return The global model θ = ( $).
Algorithm 2
The Client Updates for FedMP
Input: The index of client p, the current round number r, the downloaded models S1, S3 and the hyperparameters of Algorithm 1.
Output: The local updated model O,-
Client(p, S,. S2, S3):
2: if < Rut then
3: vir. pp wh, Sp + St, S2 Ss;
4: else
5:
ey Op Op
(.)r (g).r
+ S1, Sz:
6:
end if
7:
for 1 + do for each batch B Dp do
8:
9:
if
< R3 then C - Eq. 9 with B;
10:
11:
θ : [v wp: o
f wi: 67} 1VL,
12:
else
13:
L <- Eq. 8 with B:
14:
θ, {ur Op
10.
Op
(g).
{
(1)r (601/r VLL
15:
end
16:
end for
17:
end for
18:
return The local updated model Op-
24

---

D Proof of Theorem 1
In this section, we provide a convergence analysis of the proposed FedMP. First, some assumptions are introduced to help complete the following theoretical analysis.
Assumption For any p E [P), local loss function for local solution path C and local loss function for global solution path C.)) are L-smooth with respect to Q For & O and θ' the following inequalities hold:
S 1 - 1V (0) (O') & L2| llll
(17)
(18)
where L, and L2 are Lipschitz constants.
Assumption 2. The upper bound the variances of the local gradient to the aggregated mean can be given as follows
IV
(0) S ot.
(19)
1=1
-Z
(O)] < &G.
(20)
P
where dr and ôg are the constants.
Since the loss functions depend on the visible data, and the use of the swapping strategy results in the equivalent of extending the gradient from the sampling data, here the step size of r to the B communication rounds. Since only random clients are activated in each communication round, we need introduce the following lemmas before deriving the convergence analysis.
Lemma 1. If Assumption 2 holds, the upper bound the variances of the local gradient from the random clients to the aggregated mean can be given as follows
|
2(P S(P-K)
Eer
<
K(P-1) (8) 786))
(21)
PEC*
where C" is the selected clients set of the r-th round of communication.
Proof. First, give the following equation
Eer
Σ - - VC
K
PEC*
Σ (VLp(e") VC(OT)) 1|2
-Eer
K
PEC*
Eag
Σ (VL,(O') V2(0T)) 1|
(22)
R
pECr
1
Eer
Σ |VC,(0") VC(E")
K2
PEćr
Σ
Σ (VC,(0)) VL(O),, (0) vc((r)) -V)
PEćr p/E .PV
Considering that each communication round, each client is selected with equal probability. the K selection probability terms for the two terms in Equation 22 are P and K(K-1) P(P-I] Moreover, due to
25

---

[(0)= - Ex= Cp(O), the following equation holds:
-
(23)
(V) (L) -
p
=0.
So that,
P C
(Vp((0) V2(9), VL, V2(0)). (24)
p=1
p=lppp'
Therefore, the following formula can be derived
Eer
Σ
-
K
PEC
1
K
V
K2
P
K(K- Σ Σ (0) VC(O"),Z2'0) (V)))))))))))))) V(((7))
P(P 1) p=ippp'
P-K K
P.
(25)
Σ
[V) - 12
KP(P - 1) p=1
P-K 1
|V P (0") + +V vC) -
K(P 1) P p=1
P-K K 1
2 (0r) (09
+ 1 (0)) v v [2]
K(P 1) P
2(P K) K(P 1) (6) +
Lemma 2. If learning rate 1) S /EB/13++223
and Assumption 7 holds, it can be derived that
EB-1
Σ
|
(26)
EB
c=0
where 0r means the e-th stage in r-th communication round p-th client, and 67,0 Or
Proof. To begin with, need to introduce the following inequality
(a + b)2 < (1 + 1 )a² + (1+ c)b²,
(27)
26

---

where e >> 0 is positive number.
Equation 27 can be easily proved by
1 -a² - ob²
a² + 2ab + 6² <a² +
$
0 Σ a² + cb² -2ab.
(28)
4
S( 5(1/2 - - vob)2.
In each update stage, we can represent the process of updating as 6 6 p.e-1 - Therefore, based on the above, we can derive the following equation
n.c-1).
|0 D,e 0-11
= =100p p.e- 7/ VVC( - - 9712
=110 p.c- 7/ - ( 1r
<(1 + 110 p.e- (V),(('' 6
EB
+ (1 ( -7)()))))) p.c-))
<(1+ EB
(1+ 2EB 110 0-12 (1 (1+2
(29)
+ +(+ 2 (()) (6
+2 (0") - ( p.a ||(
5(1+
EB (L+ )
2EB + +EBB1 L 2EBB" L p.
1
0712 -01
+1+ EB
(1 + 2EB >n VC
When 7)
/ L33 the above equation can be written
10" p.c
0-12
<(1 +
- 1- +(1 + EB )1 +
2EB )n
EB
c-1
Σ (1
2it1 (1+2EB)>
EB
(1+
2001 )
(1+ 1
EB
EB
(30)
(1
(1 EB 20+1
5(1+ (1++
E
=F + EB
2e++ ("', 1
=EB(1 EB)
EB
27

---

Therefore, we can derive that
EB-
Σ 10
1
Σ
(1
EB
EB
EB
5(1 EE)
EB
2 EB
+
E
E² + EB)
1
( (1 +
YEBB
(31)
2FB +1
FB
EB(1 + EB VL, EB
22BB
2
1
<E B2 [+")(
lim (1+ EB-xc
)2BB
EB
$8E BSVEC
Theorem 1. Suppose Assumption and hold, the convergence property of the proposed method can be described by
R- R
E
se,
(32)
-=
where (-(()))L((1)
+ S, and
Hn
H. = BE
2n + 722
32 + 222)
2
(33)
F.B
641 + 4
==
B +122
K
(34)
P - K
p- (6) + +22
From Theorem Theorem
we can know that the proposed FedMP can reach the convergence with
appropriate hyperparameters choices.
Proof. First of all, based on the above lemmas and assumptions, we can derive the following equations
r++1
Σ
O
PEC-
-r 7
V Cp ( p.o.
(35)
K
P
c=0
DEB
EB-1
=0"
Σ Σ
CO, p.o) = = -'E''"''
EBK
PEC
e=0
and
VL( VL(O)
1 1 Σ (
(C)
P
p=1
(36)
p Σ V (1) ( + V (0) (6') (C.0(
1
+(41+66)110- 0||
28

---

Furthermore, Equation ca
can also be written as the second order Taylor expansion form
(()) S (()) + e', V((()))) 1 + + | 01|
(37)
Hence, the upper bound of global optimization objective for each round can be derived as
Ee- [C(e-+))- [(()))
SBc. [(ort1 - 0", v2(0"))] + L1 +
Be. [10r-1-69
2
-nEBBer 9(-9", V2(60)) + L1 + 7 Ber
2
= (VD(O)) 9" V VL(G))
(L1 + 7122 Be ||
-EEBEC [I || nEBBEr 1 - VL(e"), V((eMM) F/ F + 712)
(38)
Be- |1|95|11
nEB Ee. [IV2(E
BEB Ee- |||0 VL(
E2 + L2)
Ber |||9" VL((T) VC()))
BEB [1 11-2nBB(L L2) [[][C())]]]]
nEB
+ +1, B2 + 712) Ber |||l" - V
2
here, Ec- [llgr VL(O") can be written
Eer
Il|g"
VL(
EB-1
EB-1
=Eer
Σ Σ (.p(()
Σ Σ (0))
EBK
EBK
nEC
DEC a=0
Σ
Σ -
EBK
pEC
EB-1
Σ Σ (VL,(0)))) - (C,()"))
<2Eer
EBK
DEC e=
(39)
+ 2I
| Σ VLL(0)) VL(0))12
PEC-
EB-1
1
<2Ee'
(L1 + 72) 1/12
K
EB
PEE
+ 21E
||
Σ (C,(0)) - v V(
PEćr
4(P - K)
<16E 1 + 22rr
Σ
VC (0r))
(5), (626))))))
K
K(P -1)
PEC
29

---

<16E2B²1)(L + T7
Σ |VL,(00) V2(0)) - V2(07)1
A(P-K) K)
K(P-1) (6), +
<32E (2) +7 [IV2((")
K) K (P- (6) + +7)))
4 (P-K)
B2 >22)²
Therefore, Equation 38 can be written
Eer [C(@rt]) -
7EB
$ [1 FTL -2) +
2
-64E3 12t L12) Ee [IV2()))]1)
(40)
EE B + )
4
644 (2E + 722
K
(P-K) K)
(02 +7)
1P
We let
H BEB
[1 2n +12) 32n/ E (L TL2)²
2
(41)
E3 722)
and
1E
S
+
E B² (L1 + x22
64n2E" " B2(
+ 7L2) K
2
(42)
P K
P-1 (02 + +22)))
So Equation 40 can be written as
Eer [C(0r+1)- [()")] HEEe [[ +
(43)
When summing all communication rounds, Equation 43 can be expanded
[IVC()"))]] B[C(e") (006)))-((0)))) [()
S.
(44)
HR
E
Supplementary Experiments
E.1 Training Setup
The distributions of the train set labels each client under different levels of data heterogeneity visualized in Figuure The partitioned train sets are placed on the clients and the test sets are placed on the server for testing the global model.
Other settings: ResNet-18 [5T] used for the experiments. We implemented the proposed method using Py Torch 1.1 and deployed it on a machine configured with AMD R9 5900X, 64GB memory, and Nvidia RTX3090. For the CIFAR-10 and CIFAR- 100 datasets, we train clients and the batch sizes of the data are set to 128 and 64, respectively. For the Flowers 102 dataset, we use a system with clients and the batch size of the data is set to 64. 50% of the clients are chosen in each round for training and aggregation, and the number of local epochs E is set to 5.
system with 20
30

---

Class ID
C189 10
325
30
Client ID
Client ID
Client ID
(a) CIFAR-10, 10, a = 0.3
(b) CIFAR-10, a= 0.5
(c) CIFAR-IO, a 1.0
200
156
Class
88
0.0
00
Client
Client
Client ID
(d) CIFAR-100, a= 0.3
(c) CIFAR-100, a 0.5
(I) CIFAR-100, a= 1.0
Class
Class
Class
Client ID
Client ID
Client ID
(g) Flowers 102, 0.3
(h) Flowers 0.5
(i) Flowers102, a 1.0
Figure 8: An illustration for the distributions of labels in the train set at each client. The above figures are obtained under three heterogeneous levels the CIFAR-10, CIFAR-100 and Flowers datasets, respectively.
We set the balance round to 5, for the CIFAR-10 dataset we set the initial training round R4 to 5, and for the CIFAR-100 and Flowers102 datasets set the initial training round Ra 10. To speed up the initialization progress, we constrain the Lipschitz smoothness for the initial model. The hyperparameter y is components of the network are listed Table 5) where ConvT means the transposed convolution operator, i.e. Deconvolutional Networks [22] Under the above settings, run the proposed method to generate the final results.
0.1, and SGD optimizer with learning r 0.05 is used. The main
E.2 More Results
In order to verify that whether the proposed method works properly, output the test accuracies of the global model produced by FedMP during the training process and show it in Figure 9 with the results of the comparative methods. It can be seen that the proposed method can make the global model output a more stable test accuracy after a period of training. Moreover, the proposed method achieve the higher classification accuracies than the other comparative methods.
The results for more clients are displayed in Figure 10 Similar to the results Client obtained via the multiple paths solving approach are able to obtain better accuracy on the data at other clients, which demonstrates that the multiple paths solving approach is effective preserving global information and reaching soft consensus between the global and the local.
models
31

---

Gleesitcatoo ADC (%)
Clessmsatonn ACC (%)
ClassincaToo
Tmining Pracess (%)
Training Process (%)
Training Pracess (%)
(a) CIFAR 10, 0.3
(b) CIFAR-10. 0.5
(c) CIFAR-IO, 1.0
Classfication ACC (19)
Claasfccation
Trinning Pracess (%)
Training Process (%)
Training Pracess (%)
(d) CIFAR-104 0.3
(e) CIFAR-100. 0.5
(I) CIFAR THEAR--00,, 1.0
Classfriatton ACC (%)
Clanefircation ACC (%))
Classificatio
Training Process (%)
Training Process (%)
Training Process (%)
Flowers 102, 0.3
(h) Flowers 102, 0.5
(1) Flowers = 1.0
Figure 9: Classification accuracy curves of the global model on the test set with the training process. It can be seen that the proposed method is able to converge all dataset settings.
32

---

Table Network architecture for three test datasets. The gray background represents the parameters that need to be communicated.
(a) Network architecture for CIFAR-10.
Layer Kerne / Siride
Ouiput shape
# Params
Path-local
Path-glohal
ConvT 3x333
ConvT/3x3/5
64x96x96
1.7kx2
ConvT 34x34/2
C3R/T/34x34/2
3x224x224
0.2Mx2
ResNe--88----
1,000
11.7M
FC/-1-
100
0.IM
FC/-/-
10
1k
(b) Network architecture for CIFAR-100.
Layer/
Kernel Stride
Oulput shape
# Params
Path-local
Path-global
Conv'T /3x//3333333
Conv'T 3x3/3
64 696
1.7kx2
Conv'T 34x34/2
Cony'T /MMT/344x34/2
xx2
0.2Mx2
ResNet-18/-1-
1,000
11.7M
FC/-1-
100
0.IM
FC/-/-
100
10k
(c) Network architecture for Flowers 102.
Layer / Kernel Stride
Output shape
# Params
Path-local
Path-global
Conv 3x3/1
Conv /3x3/1
64x222x222
1.7kx2
ConvT /x
/
3x224x224
1.7kx2
ResNet-18/-1-
1,000
11.7M
FC/-/-
0.1M
FC/-/-
102
10.2k
We show the detailed results CIFAR-100 in Figure 11 The curves represent the performance of the models achieved on the local dataset after each communication round of local training and global aggregation in turn during the training process. The fluctuation of the curves represents the characteristic that the model oscillates between global and local features during the federated training process. After local training, the local features dominate and thus, the performance dataset is improved. Whereas, after aggregation, global features gain dominance and hence, the performance the local dataset decreases. The curve of using the multiple paths solving approach has less fluctuation because the model obtained with it seeks consensus among global and local features at the local training stage, therefore, although cannot fit the current dataset better, after aggregation, the model obtained will not lose too much local knowledge because the local training on other clients also takes into account the reaching of consensus. Models trained by multiple paths solving methods have smaller training fluctuations. which means that there is less feature struggle during training and the federated process will be able to perform more efficiently. Using the correct multiple paths can help the local model to reduce the vanishing of local features and keep more global features.
the local
E.3 FedMP with Full Multiple Paths
Here, instead of using the communication cost reduction method proposed in Section 2.4 of the main body, we use multiple paths solving method for the complete task network. We use the complete task network here as the mapping with global feature preferences and the mapping with local feature preferences, respectively, and use the same shared projection as in the experiments in the main text to motivate a soft consensus. The network structure is shown in Table 6) We show the classification accuracy results in Table allnn with the results in the main text. It can be seen that the results obtained are further improved after using the full multiple paths (FedMP-F), However, penalty for this enhancement would be higher communication cost, and we believe that small amount of performance degradation associated with the reduction in communication cost is acceptable. Therefore, there flexibility to trade-off between the two approaches depending on the requirements.
33

---

ACC (%)
Client ID
ACC (%)
Client
ACC (%)
Client ID
of Data
100
of Data
100
of Data
10
80
10
10
60
60
60
2.1
Multiple paths
Single path
Multiple paths
Single path
Multiple paths
Single path
(a) Local model of Client 2 Client ID of Data
(b) Local model of Client
(c) Local model of Client 4
ACC (%)
ACC (%)
Client ID of Data
ACC (%)
Client ID
100
100
of Data
10
10
10
60
60
60
Multiplc paths
Singlo path
Multiple patlss
Singlo path
Multipic paths
Single path
(d) Local model of Client 5
(e) Local model of Client
(1) Local model of Client 7
ACC (%)
Client ID of Data
ACC (%)
Client ID of Duta
ACC (%)
Client ID
100
100
100
of Data
10
10
60
60
401
401
401
4
in Multiple paths
Single path
Multiple paths mn Single path
Multiple paths
Single path
(g) Local model of Client 8
(h) Local model of Client 9
(i) Local model of Client 10
Figure 10: Classification accuracies of each local models on the data in different clients. The local model trained by the multiple paths is able to achieve better performance on the data of other clients.
Table 6: Network architecture of FedMP-F. The gray background represents the parameters that need to be communicated.
Layer/ / Kernel / Stride
Output shape
# Params
Path-local
Path-global
ResNet-18/-11 /- ResNet-
1,000
x1.7Xx2
FC/-/-
100
0.IM
--
10/ 100
1 10k
34

---

0.8
0.6
ACC
0.4
Single path
Multiple paths
10
20
25
30
35
40
Communication round
Figure 11: Accuracy curves of the models. The two sides of the horizontal axis ticks are the classification accuracies of the local trained and global aggregated models on the data of current client, respectively. The multiple paths approach achieves less performance degradation and softer fluctuations, with slighter vanishing of local features.
Table 7:
The results of CIFARs datasets for all methods. Bold and underlined results the best and the second best.
CIFAR-10
CIFAR-
Method
o=0.3
a=0.5
o=1
a=0.3
o=0.5
q=1.0
FedAvg
80.66
83.34
87.46
57.25
59.13
6T.34
FedProx
81.16
83.67
88.04
56.37
57.77
62.36
SCAFFOLD
80.60
86.08
87.66
58.65
60.96
61.81
FedNova
83.41
84.93
86.41
55.46
55.02
56.67
FedDF
78.70
76.95
88.65
52.82
52.46
59.32
MOON
82.47
85.46
87.83
57.52
59.30
62.85
FedASAM
68.98
71.59
74.44
46.84
47.91
48.52
FedPVR
84.65
85.73
87.86
59.32
60.89
62.75
FedUCS
83.61
87.79
89.43
62.19
63.24
64.28
FedMP
85.37
88.96
88.31
61.43
62.04
62.95
FedMP-F
86.92
88.10
90.34
63.64
65.10
65.39
E.4 Running Time
The running time of all methods is shown in Figure 12 For the proposed method, full multiple paths solving method is adopted to obtain fairer comparison. Even when the proposed method adopts the multiple paths solving method, it can still achieve better performance at the cost of small increase running time. We believe that this cost worth
2500
2000
1000
500
Flowers102
FedAvg
FodNova
Figure 12: Running time of different methods on three datasets.
35


Reviews:
[{"review_id": "tWNXMIBmju", "reviewer": "Reviewer_eSzu", "summary": "A multi-path solution method is proposed to combine global and local features in federated learning. This method achieves soft consensus among multiple paths, avoiding the previous approaches of using global information as the sole criterion. Considering the dominance of local features in local training, the authors employ an swapping strategy to weaken the local solution path. They adopt a data-adaptive approach to reduce the amount of model parameter transmission, thereby minimizing additional communication overhead.", "questions": "i. Can the proposed method be applied to other models or tasks beyond image classification?\n\nii. Does the multi-path method employed in the paper enhance the final model's capabilities by increasing the number of model parameters?  \n\niii. How is the data adaptation method described in the paper specifically implemented to reduce model communication overhead?  \n\niv. If two networks are trained together under the same loss function, can this be considered multi-path learning?  \n\nv. Is the proposed multi-path learning scheme unrelated to federated learning?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\ni. The proposed multi-path solution method is interesting, and experiments are conducted to demonstrate its feasibility.\n\nii. A case study is provided at the beginning to justify the necessity of the proposed method.\n\niii. The authors explain the multi-path method to distinguish it from other methods.\n\nWeaknesses\n\ni. The validation experiments seem to be limited to image classification, lacking other types of tasks.\n\nii. The paper lacks hyperparameter analysis, such as analysis of path balancing.\n\niii. Some descriptions in the paper are unclear and require further refinement.", "was_revised": true, "final_justification": "With the improvements provided in their response, my concerns have been adequately addressed.", "rebuttal": "Thank you very much for your professional comments. We will respond to each of your concerns as follows:\n* **Response to Weakness (i):** The goal of our proposed method is to seek a better global model in federated learning, and the use of image data is merely a way to validate the effectiveness of our method under federated settings. A suitable federated approach should not be restricted to a specific data type. Our original intention is to develop a method for obtaining a better global model under federated requirements, and thus we use image data as a case for validation. Similar to image classification, many tasks fundamentally involve learning data embeddings and modeling target probability distributions; hence, our method can be easily extended to tasks on other datasets. We have conducted additional experiments on the Vehicle Sensor dataset, a vehicle dataset containing 23 groups of sensor data, where each sensor group serves as the data for one client. This experiment will be included in the appendix. The following table presents the results of our method compared with several baselines:\n| Method   | FedAvg | FedProx | SCAFFOLD | FedPVR     | FedUCS | FedMP       |\n|----------|--------|---------|----------|------------|--------|-------------|\n| ACC      | 86.33  | 86.12   | 87.14    | 87.71    | 87.53  | **87.93**   |\n\n* **Response to Weakness (ii):** We will add a line plot showing the impact of the path balancing hyperparameter on the performance of the global model. Below is the table for the hyperparameter analysis experiment in CIFAR-100 dataset.\n| $\\gamma$     | 0.01  | 0.05  | 0.1   | 0.5   | 1     | 5     | 10    | 50    | 100   |\n|--------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| $\\alpha$=0.3 | 60.76 | 60.46 | 61.43 | 60.17 | 59.22 | 2.57  | 23.92 | 1.00  | 1.00  |\n| $\\alpha$=0.5 | 61.35 | 61.13 | 62.04 | 60.91 | 60.18 | 9.19  | 16.33 | 1.00  | 1.00  |\n| $\\alpha$=1.0 | 63.10 | 62.94 | 62.95 | 62.89 | 62.76 | 19.21 | 14.50 | 1.00  | 1.00  |\n\n* **Response to Weakness (iii):** We will revise or supplement the relevant descriptions based on your feedback.\n\n* **Response to Questions (i):** As mentioned in our response to Weakness(i), the objective of our proposed method is to improve the global model in federated learning, and the use of image data is only one way to verify the effectiveness of our method under federated settings. A proper federated method should not be limited to specific data types. Therefore, we have added an experiment on an additional data type. As shown in the above table, our method still achieves a certain level of performance improvement.\n\n* **Response to Questions (ii):** Here we would like to clarify that, for the multi-path method, the inference structure of the original model does not need to be changed. Therefore, during the final testing phase, the model used has the same architecture as the original model, and thus the number of parameters remains unchanged. The multi-path method only requires parallel auxiliary network structures during training to simulate different solution paths, which introduces additional auxiliary parameters. However, after training is completed, these auxiliary parameters are no longer involved in the model inference process. Hence, the improvement in federated model performance is not due to an increase in the number of parameters.\n\n* **Response to Questions (iii):** We connect an adapter between the task network, and the data to transform the raw data into a format suitable for the task network. For multi-path solving, we also attach a shared projection adapter after the task network to convert its output into the desired target output. In this case, the multi-path solving process is transformed from optimizing the entire model to optimizing only the adapters. As a result, only the adapter parameters need to be communicated, reducing communication overhead.\n\n* **Response to Questions (iv):** We believe the scenario you mentioned does not qualify as multi-path learning. The key of multi-path learning lies in having some distinct paths that negotiate and reach a consensus solution. Simply training under the same loss function without generating a consensus solution does not constitute multi-path learning. In our proposed multi-path construction, two parallel networks feed into a shared projection module, which maps their outputs into the target space (i.e., the desired output). This projection can be viewed as a consensus reached through the collaboration of the two parallel networks, with each network representing a separate solution path. Only approaches that involve such a negotiation process can be considered the multi-path learning.\n* **Response to Questions (v):** The multi-path scheme is a general idea for finding consensus between two solution paths. How to concretely implement such a multi-path solution within the federated learning framework requires further exploration. In this paper, we combine a global-preference path with a local-preference path to achieve a soft consensus, which addresses the conflict between global and local knowledges in federated learning. Therefore, this constitutes a federated learning method based on multi-path learning."}, {"review_id": "f1fHz6rwlT", "reviewer": "Reviewer_S2r7", "summary": "To address the data heterogeneity challenge in federated learning, this paper proposes a FedMP framework. This method optimizes multiple solution paths that prefer global and local features and allows them to negotiate to produce a better solution. To balance the path strength, the paper introduces a swapping strategy and uses data adaptation to reduce communication overhead. Comprehensive experiments show that this method outperforms existing advanced methods under heterogeneous data.", "questions": "I suggest that the authors provide the fundamental rationale and conceptual clarity of the \"local preference swapping strategy\", analyze the performance bottlenecks that may be caused by data adaptation, and the choice of loss function. For details, see the previous \"Weaknesses\".", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The article is well-structured, the problem and motivation are clearly described, and the proposed method is easy to understand.\n2. The proposed method is innovative, and it forms a complete solution by coordinating multiple paths with different feature preferences through a shared projection layer, combined with swapping strategies and data adaptation.\n3. The authors provide comprehensive experimental results to demonstrate the effectiveness of the method and each module.\n\nWeaknesses\n1. In local preference swapping strategy, the local model of the client is not the model trained locally in the previous round, but a local model trained by another client randomly downloaded from the server. Although ablation experiments prove its effectiveness, please explain its rationality, which seems counterintuitive. Besides, the local model of client A may come from client B in the previous round. The paper claims that this path represents \"local feature preference\". However, for client A, it is actually using the local preference model of client B. Directly describing the preferences of other clients as local preferences may cause ambiguity.\n2. In multiple paths data adaptation, to reduce communication costs, FedMP keeps the full task network constant and transfers the optimization objective from a large network to a data adapter. Will this limit the optimal model performance?\n3. I would like to know what loss function L_p used in the experiment", "was_revised": true, "final_justification": "The rebuttal has addressed my initial concerns. The explanation for the local preference swapping strategy is clear. Besides, the new ablation study addresses my question regarding the performance limitations. These clarifications have resolved my main concerns, and I have raised my score accordingly.", "rebuttal": "Thank you very much for your questions and professional comments. Below are our responses:\n* **Response to Weakness (1):** In our experiments, we found that if the swapping strategy is not adopted and the multi-path solving approach is used alone, one of the solution paths would continuously receive local data, leading to overfitting. This significantly degrades the performance of the multi-path method, as demonstrated in our ablation studies: without the swapping strategy to weaken the local-preference path, this path becomes overly dominant and unbalanced, which is an incorrect usage of multi-path learning and results in performance degradation. Therefore, we adopt the swapping strategy to balance the influence of local data on the model. The multi-path method requires two solution paths that optimize from different initial states toward the final models, which can be regarded as two distinct solution paths. If the initial state is close to a suitable value (i.e., derived from actual data rather than random initialization), it reduces the optimization difficulty of the model. Hence, we use the swapping strategy to obtain reasonable initial values from other clients for constructing the multi-path setup. Since this path originates from a model trained on a specific client, we distinguish it from the initial state derived from the global model by referring to it as a local feature preference. This terminology is introduced solely to differentiate the two solution paths. The term \"local feature preference\" indicates that, unlike the global model, this model has only been exposed to limited data, preserving unique information that has not yet been overwritten or forgotten by other data. This provides a unique interpretation of the current client's data, and this path is expected to reach consensus jointly with the global model, helping to avoid the mediocre solutions that may arise from relying solely on the global model. In summary, the swapping strategy is employed to ensure that, while maintaining multiple solution paths, the starting points of these paths are meaningful and appropriate.\n\n* **Response to Weakness (2):** Yes, it does. Therefore, we provide experimental results without data adaptation, in which the task network is fully optimized using the multi-path method, achieving better performance. Below is a comparison between the full multi-path version and the method described in the main text. The detailed experiment will be updated in the paper.\n\n| Method  | CIFAR-10 ($\\alpha=0.3$) | CIFAR-10 ($\\alpha=0.5$) | CIFAR-10 ($\\alpha=1.0$) | CIFAR-100 ($\\alpha=0.3$) | CIFAR-100 ($\\alpha=0.5$) | CIFAR-100 ($\\alpha=1.0$) |\n|---------|--------------------------|--------------------------|--------------------------|----------------------------|----------------------------|----------------------------|\n| FedMP   | 85.37                    | **88.96**                | 88.31                    | 61.43                      | 62.04                      | 62.95                      |\n| FedMP-F | **86.92**                | 88.10                    | **90.34**                | **63.64**                  | **65.10**                  | **65.39**                  |\n\n* **Response to Weakness (3):** Here, $L_p$ denotes the loss function required for training the task on a client. For the image classification experiments in this paper, we use cross-entropy as the loss function."}, {"review_id": "JQnVmRYSKY", "reviewer": "Reviewer_jWBm", "summary": "This paper proposes a federated learning method based on multi-path solution, which seeks a consensus between global feature preferences and local feature preferences through a multi-path approach, avoiding the hard consensus typically achieved by federated learning methods for heterogeneous data. The method balances and jointly optimizes global and local models to construct the aggregated model.", "questions": "1. Please respond to what I mentioned in my weaknesses. \n2. Could you further clarify the derivations and motivations behind Equations (3)–(7)? While the supplementary material (Section B) effectively highlights the novelty of the multi-path approach, the main text requires more detailed explanations for these Equations.\n3. As noted in the weaknesses, there is no discussion on the balance between the two paths. I observed hyperparameters in the multi-path optimization, but the paper lacks a sensitivity analysis on these parameters. Could you address this?\n4. In Table 2, the performance appears to degrade when using the multi-path method, which contradicts the conclusions drawn in Figure 3. What explains this discrepancy?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper describes an interesting multi-path solution method. Negotiating between global and local preferences to reach a consensus appears to be an effective solution.\n2. The authors provide theoretical proofs in paper.\n3. Extensive experiments demonstrate the effectiveness of the proposed method, including comparative studies to showcase the superiority of the multi-path method.\n\nWeaknesses\n1. Sections 2.3 and 2.4 contain excessive mathematical notation without a summary table clarifying their meanings.\n2. The methodology for data adaptation is not sufficiently elaborated.\n3. The discussion lacks an in-depth analysis of the balance between the two paths. Some explanations are unclear, for instance, the relevance of Equations (6) and (7) to the multi-path method is not explain well.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your professional review. We will respond to each of your comments as follows:\n* **Response to Weakness (1):** We will summarize all symbols used in the paper in a table, which will be added to the appendix. Additionally, each symbol will be explicitly explained at its first appearance in the text.\n| **Symbol** | **Meaning** |\n|--------|--------|\n| $\\mathcal{D}_p$ | The dataset of the $p$-th client, which is a subset of the global data |\n| $\\|\\cdot\\|$ | The number of elements in a set |\n| $\\hat{\\theta}$ | Global model parameters |\n| $\\theta_p$ | Model parameters on the $p$-th client |\n| $\\mathbf{X}^i_p$ | The $i$-th data sample of the $p$-th client |\n| $\\mathcal{L}_p^{(l)}$ | The loss function for the local solution path on the $p$-th client |\n| $\\mathcal{L}_p^{(g)}$ | The loss function for the global solution path on the $p$-th client |\n| $\\gamma$ | The path balancing hyperparameter, used to balance the influence between local and global paths |\n| $\\Phi_p^{(l)}(\\cdot)$ | The mapping of the local solution path on the $p$-th client |\n| $\\Phi_p^{(g)}(\\cdot)$ | The mapping of the global solution path on the $p$-th client |\n| $\\Psi_p(\\cdot)$ | The shared projection on the $p$-th client |\n| $\\phi^r_p$ | The model used by the $p$-th client in the $r$-th communication round to approximate $\\Phi^{(l)}_p(\\cdot)$ |\n| $\\psi_p$ | The model used by the $p$-th client to approximate the projection $\\Psi_p(\\cdot)$ |\n| $p^{r'}$ | A random sample of client indices that participated in the previous round of training |\n| $\\Omega_p(\\cdot)$ | The task network on the $p$-th client |\n| $\\tau_p^{(l),i}$ | The adapted data generated on the $p$-th client of the $i$-th data sample through the local path mapping |\n| $\\tau_p^{(g),i}$ | The adapted data generated on the $p$-th client of the $i$-th data sample through the global path mapping |\n\n* **Response to Weakness (2):** Data adaptation, as an auxiliary strategy of the multi-path method, helps reduce communication overhead. It employs adapters to transform data into a representation space recognizable by the task network, and applies the multi-path method to obtain better solutions for the adapters. In this way, only the adapters need to be exchanged during communication, rather than the entire model.\n\n* **Response to Weakness (3):** We will add a hyperparameter analysis in the appendix. The following table shows the impact of different path balancing hyperparameters on the final performance of the global federated model on the CIFAR-100 dataset. Typically, we set the hyperparameter to 0.1, which aims to balance the two paths as much as possible, with the local preference path serving as the main path and the global preference path as the auxiliary. This is because the shared projection is transmitted along with the local preference path, making it natural to treat the local preference path as the primary one. Equations (6) and (7) describe the initial states of the local preference path models at the beginning of each local training round, which constitutes a formal description of the swapping strategy, indicating that at the start of each local training, the local preference path is initialized with a randomly selected client’s model.\n\n| $\\gamma$     | 0.01  | 0.05  | 0.1   | 0.5   | 1     | 5     | 10    | 50    | 100   |\n|--------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| $\\alpha$=0.3 | 60.76 | 60.46 | 61.43 | 60.17 | 59.22 | 2.57  | 23.92 | 1.00  | 1.00  |\n| $\\alpha$=0.5 | 61.35 | 61.13 | 62.04 | 60.91 | 60.18 | 9.19  | 16.33 | 1.00  | 1.00  |\n| $\\alpha$=1.0 | 63.10 | 62.94 | 62.95 | 62.89 | 62.76 | 19.21 | 14.50 | 1.00  | 1.00  |\n\n\n* **Response to Questions (2):** These equations are indeed customized modifications of the multi-path scheme within the federated learning setting. The swapping strategy is a mechanism designed to maintain balance between the two paths in federated learning, preventing one path from becoming overly dominant. Equations (3), (4), and (5) describe the initial state of the model on client p without the swapping strategy, while Equations (6) and (7) describe the initial state on client p when the swapping strategy is applied.\n\n* **Response to Questions (3):** As mentioned in our response to Weakness (3), we have added the experimental results for this hyperparameter analysis.\n\n* **Response to Questions (4):** Indeed, when only the multi-path method is used without the swapping strategy, performance degrades. This precisely highlights the importance of correct multi-path construction for achieving good results. Without the swapping strategy, one path corresponds to a local model trained repeatedly on local data without communication, making it prone to overfitting on that local data. Consequently, during multi-path learning, this model would provide misleading or inappropriate guidance to the global model. Since our goal is to obtain a generalizable model that performs well across all clients, overly strong local bias is harmful to generalization. This is exactly why we adopt the swapping strategy. With swapping, we can still introduce local information that differs from the global model, using a model derived from additional local data as the starting point for one optimization path, thereby enabling a proper multi-path construction. In summary, providing an incorrect path can degrade performance even when using a multi-path framework. Hence, additional strategies are necessary."}, {"review_id": "8kVg5hmuZR", "reviewer": "Reviewer_Ky8Q", "summary": "The authors propose a multi-path neural network optimization method from the perspective of global and local soft consensus in federated learning, combining global and local paths to seek their consensus solution. The authors also use swap strategy and data adaptation method to further reduce additional communication overhead. The effectiveness of the proposed method is validated through extensive experiments.", "questions": "a. Does the proposed multi-path method require two identical networks to simulate different solution paths? Would this introduce additional network parameters? Could this lead to an unfair comparison when contrasting with other methods?\n\nb. How do you balance global and local paths? Is there any related hyperparameter analysis?\n\nc. What does Figure 9 mean? Please clarify further.\n\nd. What does the binding and swapping in Figure 2 specifically represent? I did not find an explanation for this binding operation in the paper.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\na. The idea of using multi-paths to find a consensus solution is insightful.\n\nb. The motivation statement is clear, and the paper provides a detailed explanation of the differences between multi-paths and other similar methods.\n\nc. The paper validates the effectiveness of the proposed framework through a variety of experiments. \n\nWeaknesses\n\na. The multi-path method appears to introduce additional computational overhead.\n\nb. Balancing different paths introduces additional hyperparameter, and there is no related hyperparameter analysis.\n\nc. The swap strategy may lead to additional information leakage.", "was_revised": true, "final_justification": "Thank you for the authors' reply. Most of my concerns have been addressed, and after reading the other reviewers' comments, I would like to maintain my score.", "rebuttal": "Thank you for your professional review. Below are our responses to your comments:\n\n* **Response to Weakness (a):** Yes, the multi-path method introduces additional optimization for the extra path. However, we adopt the data adaptation strategy to reduce this additional computational and communication overhead. Moreover, the proposed method improves the performance of federated training and provides a novel perspective for federated learning, which is of exploratory significance. Future work can focus on further reducing the additional computational cost to mitigate this issue.\n\n* **Response to Weakness (b):** Balancing different paths does introduce an additional hyperparameter. However, we can simply set it to 0.1 to avoid complicated hyperparameter tuning. We will include a hyperparameter analysis in the appendix. Below is the hyperparameter analysis result on the CIFAR-100 dataset.\n\n| $\\gamma$     | 0.01  | 0.05  | 0.1   | 0.5   | 1     | 5     | 10    | 50    | 100   |\n|--------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| $\\alpha$=0.3 | 60.76 | 60.46 | 61.43 | 60.17 | 59.22 | 2.57  | 23.92 | 1.00  | 1.00  |\n| $\\alpha$=0.5 | 61.35 | 61.13 | 62.04 | 60.91 | 60.18 | 9.19  | 16.33 | 1.00  | 1.00  |\n| $\\alpha$=1.0 | 63.10 | 62.94 | 62.95 | 62.89 | 62.76 | 19.21 | 14.50 | 1.00  | 1.00  |\n\n\n* **Response to Weakness (c):** The swapping strategy indeed carries a potential risk of additional privacy leakage, but this risk is under a controllable level. First, due to the use of data adaptation, the swapping strategy does not require transmitting the full model; thus, the amount of leaked information is limited. Second, since each swap is randomly assigned by the server, it is difficult for clients to infer gradients from the two received models. Furthermore, explicit information such as raw data or label distributions is not directly revealed. Additionally, many existing approaches focus on privacy preservation in federated learning (e.g., differential privacy), and we provide relevant discussions on these methods in the paper.\n\n* **Response to Questions (a):** Yes, the proposed multi-path method requires two networks to simulate two solution paths, which introduces additional parameters during training. However, during model inference, the network architecture and the number of parameters are identical to those of the original model without multi-path learning. Multi-path learning serves only as a training-time enhancement, effectively adding an auxiliary network in parallel to the original network. During training, the two networks optimize from different starting points, and a shared projection ensures consistency in the solution. At inference time, the auxiliary components are discarded, and only the original network is retained, thus no additional parameters are introduced in the final deployed model.\n\n* **Response to Questions (b):** As addressed in our response to weakness (b), we provide the hyperparameter analysis in the appendix. In general, maintaining a balance between the two solution paths is essential for achieving correct multi-path learning outcomes. Moreover, under the swapping strategy, the shared projection is passed along with the local preference path, meaning this path has higher compatibility with the projection. Slightly increasing its weight helps stabilize the multi-path learning process.\n\n* **Response to Questions (c):** Figure 9 shows the model’s accuracy (ACC) on local data before and after local training. Each communication round index on the horizontal axis represents two time points: the time points after model aggregation and after local training. As shown in the figure, compared to single-path learning, multi-path learning requires reaching a soft consensus between paths, making it harder to fit local data—this is reflected in the lower ACC after local training. On the other hand, because the model actively seeks a soft consensus between local and global features during training, more local knowledge is preserved during aggregation across clients. This figure illustrates that multi-path learning achieves a better balance between global and local information in federated learning, leading to an improved global model.\n\n* **Response to Questions (d):** \"Binding\" means that the shared projection and the main path form a complete and paired solution. During the swapping strategy, this pair of models is assigned together to the same client, hence we refer to them as \" Binding.\" \"Swapping\" refers to the specific execution step of the swapping strategy."}]

Author rebuttal:

Meta-review: This paper proposes a multi-path soft consensus learning framework for federated learning, which seeks consensus between global and local features to generate a better federated model. The authors conduct extensive experiments to validate the effectiveness of the multi-path approach. During the rebuttal phase, the authors provide responses to reviewers' concerns regarding path balancing and privacy issues. After the rebuttal, the reviewers' evaluations remain positive. Hence, I recommend its acceptance.
Decision: Accept (poster)