Federated Learning over Connected Modes
Authors: Dennis Grinwald
Philipp Wiesner
Shinichi Nakajima
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Linear Mode Connectivity
Primary area: other
TL;DR: Floco addresses the challenges of statistical heterogeneity in cross-silo federated learning by leveraging linear mode connectivity.

Abstract: Statistical heterogeneity in federated learning poses two major challenges: slow global training due to conflicting gradient signals, and the need of personalization for local distributions. In this work, we tackle both challenges by leveraging recent advances in \emph{linear mode connectivity} --- identifying a linearly connected low-loss region in the parameter space of neural networks, which we call solution simplex. We propose federated learning over connected modes (\textsc{Floco}), where clients are assigned local subregions in this simplex based on their gradient signals, and together learn the shared global solution simplex. This allows personalization of the client models to fit their local distributions within the degrees of freedom in the solution simplex and homogenizes the update signals for the global simplex training.  Our experiments show that \textsc{Floco} accelerates the global training process, and significantly improves the local accuracy with minimal computational overhead in cross-silo federated learning settings.

Federated Learning over Connected Modes
Dennis Grinwald 1.2 Philipp Wiesner", Shinichi Wiesner",Shinichl Nakajima 1,2,3 1.2.3 'BIFOLD, "TU Berlin, "RIKEN Center for AIP cdennis.grinwan wiesner, nakajimaJ@tu- berl in. de
Abstract
Statistical heterogeneity in federated learning poses two major challenges: slow global training due to conflicting gradient signals, and the need of personalization for local distributions. In this work, we tackle both challenges by leveraging recent advances in linear mode connectivity - identifying a linearly connected low-loss region in the parameter space of neural networks, which we call solution simplex. We propose federated learning over connected modes (FLOCO), where clients are assigned local subregions in this simplex based on their gradient signals, and together learn the shared global solution simplex. This allows personalization of the client models to fit their local distributions within the degrees of freedom the solution simplex and homogenizes the update signals for the global simplex training. Our experiments show that FLOCO accelerates the global training process, and significantly improves the local accuracy with minimal computational overhead in cross-silo federated learning settings.
Introduction
Federated learning (FL) (1 is decentralized machine learning paradigm that facilitates collaborative model training across distributed devices while preserving data privacy. However, in typical real applications, statistical tettrooeneetyy-non-ddeeeeeee
and independently distributed (non-IID) data distributions at clients-makes makes it difficult to train well-pent rell-eerforming models. To tackle this difficulty, various methods have been proposed. c.g., personalized FL (2), clustered FL [3), advanced client selection strategies [4) robust aggregation [5]. regularization strategies [6] and federated meta- and multi-task learning approaches 7 [). These methods aim either at training a global model that performs well on the global distribution [9] or, as it is common in personalized FL., at training multiple deeendede models each of which performs well on its local distribution [10. These two aims often pose a trade-off-a model that shows better local performance tends to suffer from worse global performance, and vice versa. this work, we aim to develop a FL method that improves local performance compared to state-of-the art methods without sacrificing global performance.
Our approach leverages recent findings on mode connectiviry paths in the parameter space between independently trained neural networks-and its applications 14.. These works show that minima for the same task are typically connected by simple low-loss curves, and that this connectivity benefits training for multi-task and continual learning. In particular, the authors show that embracing mode connectivity between models improves accuracy on each task and remedies the risk of catastrophic forgetting.
the existence of low-loss
In this paper, we leverage such effects, and propose federated learning over connected modes (FLocO), where the clients share and together train a solution simplex-a linearly connected low-loss region in the parameter space. Specifically, FL.CCO represents clients as points within the standard simplex based on the similarity between their gradients, and assigns each client a specific subregion of the simplex. Clients then participate in FL by sampling different models within their assigned subregions and sending back the gradient information to update the vertices of the global solution simplex (see
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Subragion assignment at
Sampling during training 2
stacked simplex gradients
81
81
A
client
client
projection onto simplex
:
A0
client
client
02
03
82
8,
Resulting test loss across simplex
2.5
2.33
2.21
2.4
2.39
2.21
2.3
2.33
2.3
2.27
2.2
82
global distribution
By
8,
distribution at diient 1
83
8g
distribution at client 2
83
Figure 1: FLOCO expresses each client as a point (* in the top-center plot) by projecting the gradient signals onto the simplex, so that similar clients are close to each other. In each communication round, each client uniformly samples points in the neighborhood of their projected point (top-right plot), and jointly train the solution simplex. The lower row shows the resulting test loss on the solution simplex, where the loss for the global distribution (left) uniformly small, while the losses for individual local distributions (center for client and right for client 2) are small around their projected points.
Fig[pp
This method facilitates collaborative training through the common solution simplex, while allowing for client-specific personalization according to their local data distributions.
Our experiments show that FLOCO outperforms common FL baselines (FedAvg [. FedProx 15]
and state-of-the-art personalized FL approaches (FedRoD [16]. APFL [7]. Ditto [18), FedPer 19]
on both local and global test metrics-withouu introducing significant computational overhead
cross-silo FL settings. We also demonstrate additional benefits of FLOCO, including better uncertainty
estimation, improved worst client performance, and smaller divergence of gradient signals.
Our main contributions are summarized as follows:
We propose FLOCO, a novel method that trains a solution simplex for mitigating statistical heterogeneity of clients, and demonstrate its state-of-the-art performance for local personalized FI..
We propose simple projection method to express clients as points in the standard simplex based on the gradient signals, and establish a procedure subregion assignments.
We conduct experimental evaluations on semi-artificial and real-world FL benchmarks with detailed analyses of the behavior of FLOCO, which give insights into how the mechanism improves performance compared to the baselines.
We provide implementations of F1.0CO in the FL frameworks Fl.-bench 20 and Flower 21 Our code publicly available:
cow/dennis wttp://dennii-giiaaadd/de/eeee grw.iaaddwwaaloooooo
Background
In this section, we briefly explain the concepts behind federated learning and mode connectivity, which form the backbone our approach. The symbols that we use throughout the paper are listed in Table Appendix.

---

2.1 Federated Learning
Assume a federated system where the server has global model go and the clients have their local models {gx J FL aims to obtain the best performing models {9t k=o such that
K
96 argming, 9u (go) = (()
9 argming. 9r. (ga) for 1,,,.... I,..., K, where (g) ((a.))-p(a..) [g (x, u))]
Here, p(k) is the normalized population of data samples for the k-th client, P' (a, y) is the data distribution for the client K and f(g, (æ. y)) is the loss, e.g., cross-entropy, of the model g sample (æ, y) E IR' x {l,..., L). where / is the dimension of an input data sample. Global [22] and personalized [10] FL aim to approximate 90 and {- respectively, by using the training data D (Dx) k= observed by the clients. Throughout the paper, we assume that all models are neural networks (NNs) y = go (æ; wx) with the same architecture, and represent the model Ok with its NN parameters wy E R i.e., hereafter represent 9& gi(2;; (æ:; wxx by wy and thus denote, e.g.. Fi (gx) by F (wne). Let N ER=1 Ng be the total number of samples, where N₂ = 1Dxl-
For the independent and identically distributed (IID) data setting, i.c., Pk (z y) = p(z, y). Ak = 1...., K. the global and personalized FL aim for the same goal, and the minimum loss solution for the given training data is
wo Wk argmin F(w) Ek=1 N Fx(w))
(3)
where FA (ur) = N₁. E(aw)ED f(n (x.
In this setting, Federated Averaging (FedAvg)
w ++1 = + Exest
N2. N
Aw
1+ for t = I....,T.
(4)
is known converge to Wp and thus solve Eq. the (-th communication round, and Aw gradient descent,
G). Here, St is the set of clients that participate
w
c-l w/ is the update after T" steps of the local
k
2'+1 "" V for L' = 1'.
where v w/ in proximity regularization [3]. auxiliary data [24], and ensembling (25).
w0+1 and y is the step size. FedAvg has been further enhanced with, e.g..
On the other hand, in the more realistic non-lII setting, where w w FedAvg and its variants suffer from slow convergence and poor local performance [26]. To address such challenges, Ditto [8] was proposed for personalized FL, i.e., to approximate the best local models { I- Ditto has two training phases: it first trains the global model w. by FedAvg, then trains the local models with proximity regularization to Wo i.e.,
W argmin w. (w wx) = Fh (w)) + -1 - 1
where À controls the divergence from the global model. Ditto has been shown to outperform many other non-IID FL methods, including the client clustering method HYPCLUSTER, adaptive federated
learning (APFL). which interpolates between global and local models (27 Loopless Local SGD
(L2SGD), which applies global and local model average regularization 28 and MOCHA 77. which fits task-specific models through a multi-task objective.
2.2 Mode Connectivity and Solution Simplex
Freeman and Bruna (2017) 29). as well as Garipov et al. (2018) [2]. discovered the mode connec- tivity in the NN parameter space space-the existence of simple regions with low training loss between two well-trained models from different initializations. Nagarajan and Kolter (2019) [13] showed that the path is linear when the models are trained from the same initialization, but with different ordering of training data. Frankle al. (2020) 30 showed that the same pre-trained models stay mnearly-connecter after tie-ttnnig with gradient noise or different data ordering.
3

---

Benton et al. (2021) 31 found that the low loss connection is not necessarily in ID, and [3] showed that simplex.
[({0m})) ( ((00m) = = O'm Om a E AV).
(6)
within which any point has a small loss, can be trained from randomly initialized endpoints.
Here, (6m E IP ?
I are the endpoints or vertices of the simplex, and M {a
[0, 1| || t} denotes the M-dimensional standard simplex. This simplex learning is performed by finding the endpoints that (approximately) minimize
E E (pp(a))(( Eur- [[ (ær, v))]]
where 2w denotes the uniform distribution on set W. During training, one model realization wo from the simplex gets sampled and its gradient update wrt. the loss, e.g. cross-entropy, gets backpropagated to the simplex endpoints [mm] W+1
Proposed Method
In this section, introduce our approach, where the mode connectivity is leveraged for collaborative training between personalized client models.
3.1 Federated Learning over Connected Modes (FLOCO)
The main idea behind FLOCO to assign subregions of the solution simplex (6) clients in
such a way that similar clients train neighboring (and overlapped) regions, while enforcing (linear) connectivity all other client's subregions. The connectivity constraint systematically regularizes client training and allows for efficient collaboration between them.
The subregion assignments need reflect the similarity between the clients. To this end, FLOCo expresses each client as a point in the standard simplex, based on the gradient update signals. Specifically, applies the Euclidean projection onto the positive simplex [33] with the Riesz s-Energy regularization [34], which gives well spreaded projections that preserve the similarity between the client's gradient signals as much as possible. Once the clients are projected onto the standard simplex as (or E A J we assign the I.1-ball with radius P around Okk, i.e., RR = {ar E A AM; 1111=-aa S p}. to the k-th client. Note that the gradient update signals are informative for the subregion assignment only after the (global) model trained to some extent. Therefore, the subregion assignment is performed after FL rounds are performed. Before the assignment, i.e., t S T, all clients train the whole standard simplex R2 A Yk, which corresponds to simplex learning version of FedAvg.
Starting from randomly initialized simplex endpoints (0.m M++ FL.CCO performs the following steps for each participating client & S² in each communication round t:
1. The server sends the current endpoints {Ø M to the client k.
2. The client performs simplex learning only on the assigned subregion R2 as a local update.
3. The client sends the local update of the endpoints to the server.
This way, FLOCO is expected to learn the global solution simplex {wa; a E A M while allowing personalization to local client distributions within the solution simplex. Algorithm I shows the main steps.
Although the simplex learning be applied to all parameters, our preliminary experiment showed that applying simplex learning only of the parameters in the last fully fully-connected layer (while point- estimating the other parameters) is sufficient. Therefore, our FLOco only applies the simplex learning to the last layer, which gives other benefits including applicability to fine-tuning of ppe-trained models, and significant reduction of computational and communication costs, shown in Section 4.3
Below, we describe detailed procedures of client projection, local and global updates in the communi- cation rounds, and inference in the test time.

---

Algorithm 1: Federated Learning over Connected Modes (FLOCO).
Input
:number of communication rounds T, number of clients K, simplex dimension M, subregion assignment round т, subregion radius p
{0 LM+1 m m=1 f- initialize simplex(MM
2 Rx < AM V/ I.....K K // set all client subregions to the whole standard simplex
for t to T do
if = then
[{ kmm 1k=1 - collect_ aad___ndddtkkk strd_satckgrddietsss/// M
{au) 1k t client_ llweeceeeeeeeeeeeeeee representation(([AA m= 3k=1) [R + assign aaasaanssssseeeees (faregiion(((x)))==)
1M+11K
St + choose taaaus_paatccppatngchlle for k E St do
[0 M local_ local_update((().. ( J 1 Rx)
10
m.kk
m== + global_updatel (0 M1+1)
11
m, mm= J )k()))
3.2 Client Gradient Projection onto Standard Simplex
We explain how to obtain the representations {as E A } of the clients in the standard simplex such that similar clients located close each other, while all clients are well-spread across the simplex.
At communication round ( T, FLOCO uses the gradient updates of the endpoints { m. m=1 as a representation of the client We concatenate the gradients for the M + endpoints into ((M + 1) D)-dimensional vector, and apply the PCA projection onto the M dimensional space, yielding Hyg IRd as a low dimensional representation. To project {NE} onto the standard simplex A we solve the following minimization problem:
M++1
min 2>0
Σn -
(8)
subject to: Bx(2) = argmin GREAM || -Kk 12.
(9)
The objective function Eq. (8) is the Riesz s-Energy
a generalization of potential energy of multiple particles in physical space, and therefore its minimizer correponds to the state where particles are well spread across the space. The minimization in the constraint ( corresponds to
the Euclidean projection onto the positive simplex [33), which forces [ keep the locations of the PCA projections {KR} } of the clients. Fortunately, this minimization problem (for fixed =) is convex, and be efficiently solved (see Appendix A). We solve the main problem (8) by computing
3g on a grid E [0, 1] with the interval 0.001, and set the representations of the clients to (5) where is the minimizer of Eq.
O
3.3 Communication Round: Local and Global Updates
In the t-th communication round, the server sends the current endpoints (0 Im=l to the participating clients St. Then, each client E S draws one sample per mini-batch from the uniform distribution A {o+)8r URR the assigned subregion and applies T" local updates,
ot'+1
? V Fi(wa),
(10)
to the endpoints with a sequentially chosen from A Here 0
m. m.k The local
T g
updates (A0'-1- g++1
0º m= sent back the server, which updates the endpoints
e 0' Ekesi No. 10 I+1
(11)
m.k
Note, that we do not rely regularizer that forces the diversity of the endpoints, as in 132 In FLOCO, the diversity of local client distributions prevents the simplex endpoints from collapsing to single point.
5

---

As explained in Section 3.1 the client subregions are initially set to the whole simplex A M before the subregion assignment performed at r, which corresponds a straightforward application of the simplex learning to FedAvg. After the subregion assignment, F1.OCO uses the degrees of freedom within the solution simplex personalize clients models.
3.4 FLOCO
We can further enhance the personalized FL performance of FLOCO by additionally fine-tuning a local model as in Ditto [18]. In this extension, called FLOCO ", each client personalizes the global 0 endpoints {0 mm m=1 by local gradient descent to minimize the Ditto objective, i.e.. k
(0 m argmin Fx ((0 (0
= Ean [Fa(wa }1)] + M+1 10m - 0 11.
3.5 Inference
With the trained endpoints (0m = T M+1
m=1 we simply use wa ( ((0m) m=1) as the global model, where ão- M++1 1 1M+1 with ln denoting the D-dimensional all one vector. For local models, we use { (10m) ) where ā+ ag. For FLOCOt, we fine-tune the corresponding subspace regions R22 for E' local epochs.
Experiments
In this section, we experimenaally show the advantages of FLOCO and FLOco over the baselines.
4.1 Experimental Setting
Datasets and models. To evaluate method, we perform image classification the CIFAR- 10 (35] and FEMNIST (36] datasets. For CIFAR-10, we train a CNN (CifarCNN) from scratch, following 37,, I and fine-tune ResNet-18 38 pre-trained on ImageNet [39], as in 40 For
FEMNIST, we train a CNN (FemnistCNN) from scratch, as in [.. and fine-tune a SqueezeNet pre-trained on ImageNet, following [40]. We provide table with the training hyperparameters that we use for each dataset/model setting in Appendix B
Data heterogeneity for non-FL benchmarks. The FEMNIST dataset is an FL benchmark based on real data, where client heterogeneity is inherently embedded in the dataset. For CIFAR- 10, simulate statistical heterogeneity by two partitioning procedures. The first procedure by [42 partitions clients in equally sized groups and assigns each group a set of primary classes. Every client gets % of its data from its group's 's primary classes and (100- g) % from the remaining classes. We apply this method with g = 80 for five groups and refer to this split as 5-Fold. For example, in 5-Fold, 20 % of the clients get assigned 80 % samples from elasses 1-2 and 20 % from classes 3-10. The second procedure, inspired by [43] and 44]. draws the multinomial parameters of the client distributions p Multi(y; (b)) from Dirichlet, i.e., de (B), where 3 the concentration parameter controlling the sparsity and heterogeneity-4 + oc concentrates the mass the uniform distribution (and thus homogeneous), while small 0
CIFAR- 10.
generates sparse and
heterogeneous non-lID client distributions.
Baseline methods. Besides FedAvg [ and FedProx (23 for global FL, chose FedRoD [16). APFL [77 Ditto [8), and FedPer as state-of-the-att personalized FL baselines.
FLOCo Hyperparameters. For CifarCNN on the simulated non-IID splits Dir(0.3)/Five-Fol we set 250, .M 20/10, p 0.1. For FemnistCNN on FEMNIST we set T = 250, M 10, = 0.5. For pre-trained ResNet-18 8 on the simulated non-lID splits Dir(O.3)/Five-ool. we set T 50, M 20/10, = 0.1 and for the pre-trained SqueezeNet on FEMNIST we set r
250. M = 3. = 0.5. We found those settings work well in our preliminary experiments, and conducted ablation study with other parameter settings in Appendix [D] For the baselines, we follow the recommended parameter settings by the authors, which are detailed in Appendix B

---

Table 1: Average global and local test accuracy.
CIFAR-I0
FEMNIST
CifarCNN
pre-trained ResNet-18 18
FeNnistCNN
pre-trained
5-Fold
Dir(0.3)
5-Fold
Dir(0.3)
SqueezeNet
FedAvg
60.36
60.38
60.74
60.78
75.33
76.94
68.59
59.27
78.83
79.84
75.13
75.51
FedProx
60.68
60.36
60.40
60.27
76.93
77.46
62.27
60.26
78.84
80.15
75.47
75.99
FedPer
40.23
65.42
33.90
67.86
68.64
84.06
50.84
85.05
50.76
73.83
64.03
74.43
APFL
60.56
60.33
60.55
60.65
53.25
46.46
50.97
44.57
4.95
4.98
38.21
58,86
Ditto
60.36
72.22
60.74
73.90
75.33
69.18
68.59
76,23
78.83
82.02
57.89
65.06
FedRoD
56.36
74.03
46.12
76.42
17.46
31.82
10.27
33.85
4.95
4.99
4.95
4.95
FLoCo
62.93
71.78
62.57
71.04
77.15
85.90
73.62
80.38
78.99
84. 09
75.86
77.00
FLOCO
62.93
75.08
62.57
76.50
77.15
84.88
73.62
85.89
78.99
84.75
75.86
82.41
Table 2: Average global and local expected test calibration error.
CIFAR-I0
FEMNIST
CifarCNN
preitrained ResNet-18
FemmistCNN
pre-trained
5-Fold
Dir(0.3)
5-Fold
Dir(0.3)
SqueezeNet
FedAvg
24.08
25.61
22.95
24.51
13.77
19.57
13.48
19.57
12.40
16.86
15.54
20.
FedProx
23.76
25.56
23.19
24.89
12.40
12.41
15.16
19.83
12.41
16.93
15.48
20.04
FedPer
47.75
28:22
56.39
25.70
19.73
11.19
38.48
10.88
38.44
21.68
28.28
22.31
APFL
23.30
25.01
22.19
23.91
28.39
33.39
20.02
26.01
4.95
4.98
7.6
15.82
Ditto
24.08
19.13
22.95
17.64
13.77
16.43
13.48
14.50
12.40
14.65
15.54
18.06
FedRoD
29.78
18.40
41.91
17.45
75.59
64.07
89.31
64.07
4.95
4.99
4.99
4.99
FLOCO
21.82
18.44
20.06
18.75
11.48
9.44
10.30
11.28
10.28
13.94
14.65
19.15
FLOCo
21.82
17.69
20.06
16.50
11.48
12.42
10.30
11.98
10.28
13.87
14.65
15.35
Evaluation criteria. For the performance evaluation, we adopt two metrics, the test accuracy measured after the last communication round (ACC) and the tom-tto-bestacccuracy (TTA), each for evaluating the global and local FL. performance. ACC is the last test accuracy over T communication Mus
rounds, i.e, ACC(T) Nis. Emsr 1 (y: = argmax (a,;; δ )), where 1-() is the indicator function that equals to if the event is true and 0 otherwise. TTA evaluates the number of communication rounds needed
achieve the best baseline (FedAvg and Ditto in this paper) test accuracy, i.c., ACCred (7). We report TTA improvement, i.e. the TTA of the baseline, e.g. FedAvg, divided by the TTA of the benchmarked method, egg. FL.CC.. Moreover, we report the aopected-calibratinn (ECE) 45 common measure that evaluates the quality of uncertainty estimation of a trained model, for the last communication round.
4.2 Results
Table D and summarize the main experimental results, where FLOCO and FLOCO consistently outperform the baselines across the different experiments in terms global (red) and local (blue) test accuracy, well test ECE. The global and local test metries are measured after the last communication round and averaged over 5 different seed runs. The best performances are highlighted in bold, while the underlined entries indicate the settings that did not converge properly. Note that the global test performances of FEDAVG and DITTO, as well as FLOCO and FLOcO+, are the same since they use the same global model. Below we report detailed observations.
Global and local FL test accuracy. We first evaluate the global and local test performance on CIFAR- 10 with the non-III data splits generated by the 5-Fold and Dir(3) procedures, as well as the natural non-IlI data splits in the FEMNIST dataset. Table [ shows the test accuracies on CIFAR-10 with CifarCNN trained from random initialization (left) and ResNet- fine-tuned from the ImageNet pre-trained model (center), respectively. also shows the test accuracies on FEMNIST with FemnistCNN trained from random initialization (left) and SqueezeNet fine-tuned from the ImageNet pre-trained model (right). We clearly see that FLoCO and FlocO outperform all baselines in terms of average local (bluc) test accuracy by up to 6%. as well as global (red) by up 5%.
7

---

Calibration. We evaluate and benchmark the quality of uncertainty estimation of all methods. For this purpose we evaluate the global as well as average local ECE on each model-dataset combination for each baseline on the test dataset and show the results in Table 2 achieve better Expected Calibration Error (ECE) across settings, with two exceptions: training pre-trained ResNet- 18 on the CIFAR-10 10 Dir(0.3) split and pre-trained SqueezeNerV1 on FEMNIST. In the first case, the average local ECE for FLOCO and FLOCO slightly worse than that of FedPer, suggesting mild overconfident for some clients. In the second case, the next best method (APFL) yields significantly lower global test accuracy than our method, making a fair comparison their ECE difficult.
As shown, FL.OC and Fr.Oco*
Worst client performance. We evaluate the av- erage local and global test accuracies of the worst 5% of clients, a standard approach for assessing potential biases of the FL method toward specific clients or client groups [46]. The worst 5% client performance all CIFAR-I0/oode combinations is evaluated over 5 trial runs, with results shown in the table the right. We observe that FLoco achieves the highest performance among worst- performing clients across all settings, with a 17% improvement over FedAvg, and up to 1.5% over the next best baseline.
Table 3: Average local test accuracy for the 5% worst performing clients on CIFAR-10.
CIFAR-10 (CifarCNN)
5-Fold
Dir(0.3)
FedAvg
44.0 0.02
42.93 + 0.03
FedProx
43.87 + 0.02
43.23 + 0.03
FedPer
52.67 +: 0.02
51.01 + 0.02
APFL
43.27 0.02
46.36 1 0.03
Ditto
58.20 1 0.03
58.69 1 1.03
FedRoD
60.20 1 0.02
61.12 1 0.03
FLOCO
61.73 1 0.02
61.13 1 0.03
Time-to-accuracy Similar to Table T we plot the TTA improvement for FLOCO. In particular, we show the TTA improvement of F1.OCO over FedAvg and FedProx, and the TTA improvement of FLOCO over Ditto, FedPer and FedRod, as these methods include local fine-tuning. We report all TTAs in Table 4 The underlined entries indicate the cases where the test accuracies of our methods exceed the baseline method's s maximum accuracy already at the initial evaluation round, while the entries labeled x1.00' represent the instances where our methods take the same evaluation rounds to achieve the baseline method's 's maximum accuracy, i.e., comparable in terms of TTA. In addition to test accuracy, we also observe improvement in Time-to Accuuracy (TTA) for method across all settings.
Table 4: Improvements for global and local time-to-acee -ac.uurtccccrccy...
CIFAR-10
FEMNIST
CifarCNN
pre pre-trained ResNet-18
FemnistCNN
pre-trained
S-Fold
Dir(O.3)
5-Fold
Dir(O.3)
SqueezeNet
FLOCO FedAvg
x5.5
x4.6
x3.4
x3.1
x1.3
xI.8
xl.2
x8.0
x1.7
x1.2
x1.1
x1.1
FLOCO vs. FedProx
x5.1
x4.9
x3.3
x3.8
x1.0
x1.8
x1.2
x9.0
x3.0
x1.2
x1.0
x1.1
FLoco Ditto
x5.5
x2.3
x3.4
x2.1
x1.3
x2.0
x1.2
x1.7
x4.0
x4.0
FLOCo FedPer
x1.0
xI.5
x1.0
x1.3
x1.6
xl.5
x1.5
x1.5
x7
x7
x7.0
x2.7
FLocO+ vs. FedRoD
x9.4
x1.6
x24.5
x1.3
x10
x10
x10
x10
x7
x7
x10
x10
4.3
Analysis and Discussion
In this section, we provide further analyses and discussion on FLOCO.
Solution structure in simplex. First, we confirm that FLOCO uses the degrees of freedom within the solution simplex for personalization. To this end, we draw approximately 500 uniformly distributed points in the solution simplex, and evaluate the global and the local test accuracy of the corresponding models. Figure (bottom row) shows the global test accuracy (left most) and the local test accuracy (center and right) for two clients. As expected, for global test dataset the solution simplex performs uniformly well across all its area, while the losses for the two individual local client distributions are small around their projected points (x). This result indicates that the heterogeneous sharing of the solution simplex across the clients properly works as designed.
8

---

Global test accuracy
Average client test accuracy
Total gradient variance (A
0.75
FedAvg
APFL
0.75
Ditto
FedProx
0.7
FedRoD
Rocco
0.7
0.04
0.65
FedPer
Floco
0.65
0.03
0.6
0.6
0.55
0.55
0.02
0.5
0.5
0.01
0.45
0.45
0.4
0.4
0
100
200 300
400
500
100
200 300 400 500
0
100 200 300 400 500
Communication round
Communication round
Communication round
Figure 2: Global (left) and average local (center) test accuracy for CifarCNN on CIFAR-10. 10, 5-Fold. For FLOCO, we can clearly observe a jump in average local test accuracy at r = 250, which is result of our subregion assignment. Right shows the total variance of the gradients for the last fully-connected layer.
Gradient variance reduction and stability of training. Figure 2 shows the test accuracy curves during training for global (left) and average local (center) test accuracies of different methods with the standard deviation over 5 trials as shadows. We observe that FLoco and FLOCO not only converge faster than the global and pFL baselines respectively, but also show small standard deviation across trials. The latter implies that our systematic regularization through the solution simplex stabilizes the training dynamics significantly. Figure
shows the total gradient variance-the sum of the variances of the updates Auf w w for FedAvg and FedProx (which almost overlap with each other), and A0' m. = o m a! i for FLoCO, respectively. More specifically, compute the variance over the last fully-connected layer, given by
(fo = ERESt Aww
TST EkES Awt
(12)
for FedAvg and FedProx, and by
ooo M+1 IMH ExEs: ||00 m.k 18T EvEs: Aet A
(13)
We have not plotted the gradient variances of FLOco+ and the other pFL methods, since those the same as for FLOCO and FEDAVG, respectively. As discussed in [47 (8), small total variance indicates effective collaborations with consistent gradient signals between the clients, leading to better performance. From the figure, we see that the total gradient variance of FLOCO is much lower and more stable, in terms of standard deviation, than the baseline methods, which, together with its good performance observed in Table I is consistent with their discussion. The variance reduction with FLOCO implies that the degrees of freedom of the solution simplex can absorb the heterogeneity of clients to some extent, making the gradient signals more homogeneous. Moreover, [49] argued that the last classification layer has the biggest impact on performance, implying that reducing the total variance of the classification layer, FLOCO does with simplex learning, is most effective. As we show in the Appendix [ applying simplex learning to only the last layer, instead of learning a simplex in the whole parameter space, achieves faster personalized and global convergence.
Computational complexity. If the batch size is one, simplex training adds O M) computational complexity for each layer, where и is the parameter complexity of the layer, e.g., = d. T. for fully connected layer with d input and output neurons, and M is the simplex dimension [32] For FLOcO, this additional complexity only applies to the classification layer. For inference, no additional complexity arises, compared FedAvg, because inference performed by the single model corresponding to the cluster center. Since the most modern architectures, e.g., ResNet- 18 and Vision Transformer (ViT) [50], have parameter complexity of ((GFE) > O(Gc), where GFE and Ge are the complexities of the feature extractor and the classification layer, respectively, the additional training complexity, applied only to the classification layer, of FLOCO is ignorable, i.e., O(GFE) > OG M). The same applies to the communication costs: since the simplex learning is applied only to the classification layer, the increase of communication costs are ignorable compared to the communication costs for the feature extractor.

---

Related Work
There are few existing works that apply simplex learning federated learning. 37 proposed SuPerFed, which enforces low loss simplex between independently initialized global and client models. yielding good personalized FL performance. This approach builds on 27,, which finds optimal interpolation coefficients between a global and local model to improve personalized FL. However, their simplex is restricted to be ID, i.e., a line segment, and the global model performance is comparable to the plain FedAvg. Moreover, they train solution simplex over all layers between global and local models, which is computationally expensive and limits applicability to training from scratch. This should be avoided if pre-trained models available [40] [1]] Our method generalizes to training low-loss simplices of higher dimensions in a FL setting, tackles both the global and personalized FL objectives, is applicable to pre-trained models, and shows significant performance gains by employing our proposed subregion assignment procedure. In Table 7 of Appendix EE we benchmark FLOCO against the SuPerFed baseline on the CIFAR-10. 5-Fold, well as Dir(0.5) splits using both a CifarCNN trained from scratch as well as a pre- trained ResNet18 on both global as well as local test performance, where we observe that FLOco outperforms SuPerFed both in terms of global as well as local accuracy in all settings.
Limitations
In this work, we only evaluate our method on cross-silo FL settings with up to 100 clients. Unlike cross-device FL, which typically involves much larger set of stateless clients (i.e., clients with limited data that hinders reliable modeling), our approach assumes stateful clients, each with sufficient data to enable effective grouping of similar clients. While our current analysis focuses cross-silo FI., extending our method to the cross-device setting is an important direction for future research. Additionally, a thorough theoretical analysis of our approach remains a future research objective.
7 Conclusion
FL on highly non-IID client data distributions remains a challenging problem and a very actively researched topic. Recent works tackle non-llI FL. settings either through global personalized FL. While the former aims to find a single optimal set of parameters that a global objective, the latter tries optimize multiple local models each of which fits the local distribution well. These two different objectives may pose a trade-off, that is, personalized FL might adapt models to strongly to local distributions which might harm the global performance, while global FL solutions might fit none of the local distributions if the local distributions are diverse. In this paper, we addressed this issue by leveraging the mode-connectivity of neural networks. Specifically, we propose FLOCC,, where each client trains an assigned subregion within the solution simplex, which allows for personalization, and the same, contributes learning well-performing global model. FLOCO achieves wate-of-the-ar performance in both global and personalized FL, with minimal computational and communication overhead during training and no overhead during inference.
Promising future research directions include better understanding the pecision-makinge process of solution simplex training through global and local explainable Al methods [52 [4]. Furthermore, we want to apply our approach to continual learning problems and FL scenarios with highly varying
client availability (55 56]]
Acknowledgements
This work was funded by the German Ministry for Education and Research as BIFOLD Berlin Institute for the Foundations of Learning and Data (ref. BIFOLD24B).
10

---

References
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. cotmmumiiaaion-effiiienet learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282, 2017.
[2] Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference Smart Trends in Systems, Security and Sustainability (WorldS4), pages 794-797. IEEE, 2020.
[3] Felix Sattler, Klaus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE Transactions Neural Networks and Learning Systems, 3283)3370--372, 2020.
[4] Fan Lai, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. Oort: Efficient federated learning via guided participant selection. In Symposium on Operating Systems Design and Implementation, pages 19-35, 2021.
[5] Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142 142--14-,,,, 2022.
[6] Alp Emre Durmus, Zhao Yue, Matas Ramon, Mattina Matthew, Whatmough Paul, and Saligrama Venkatesh. Federated learning based on dynamic regularization. In International conference on learning representations, 2021.
[7] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in Neural Information Processing Systems, 30, 2017.
[8] Durmus Alp Emre Acar, Yue Zhao, Ruizhao Zhu, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Debiasing model updates for improving personalized federated training. In International conference on machine learning, pages 21-31. PMLR, 2021.
[9] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint arXiv:1806.00 00582, 2018.
[10] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[11] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference Machine Learning, pages 1309-1318, 2018.
[12] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in Neural Information Processing Systems, 31, 2018.
[13] Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generalization deep learning, Advances in Neural Information Processing Systems, 32, 2019.
[14] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity multitask and continual learning. In Interna- tional Conference on Learning Representations, 2021.
[15] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint 1r115:1511.07543, 2015.
[16) Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[17] Xueting Ma, Guorui Ma, Yang Liu, and Shuhan Qi. APCSMA: adaptive personalized client- selection and aggee-aagge algorithm for federated learning in edge computing scenarios. Entropy, 26(8):712 12, 2024.

---

[18] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: learning through personalization. In International Conference on Machine Learning, pages 6357-6368, 2021.
Fair and robust federated
[19] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. CoRR, abs/1912.000 2019.
[20] Jiahao Tan and Xinpeng Wang. FL-bench: A federated learning benchmark for solving image classification tasks.
[21] Daniel Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Germandez-Marquee Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusmão, and Nicholas Lane. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007. 14390, 2020.
[22] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge-Baud Based Systems, 216:106775, 2021.
[23] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Machine Learning and Systems, 2:429-450, 2020.
[24] Felix Sattler, Tim Korjakow, Roman Rischke, and Wojciech Samek. Fedaux: Leveraging unlabeled auxiliary data in federated learning. IEEE Transactions on Neural Networks and Learning Systems, 34(9):5531-5555 2023.
[25] Naichen Shi, Fan Lai, Raed Al Kontar, and Mosharaf Chowdhury. Fed-ensemble: Ensemble models in federated learning for improved generalization and uncertainty quantification. IEEE Trunsuctions on Automation Science and Engineering, 2023.
[26] Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey, Neurocomputiing 465:371-390, 2021.
[27] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.14461, 2020.
[28] Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint 18120098
[29] C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimiza- tion. In International Conference on Learning Representations, 2017.
[30] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning, pages 3259-3269, 2020.
[31] Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss sur- face simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pages 769-779, 2021.
[32] Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In International Conference on Machine Learning, pages 1 11217-11227, 2021.
[33] Mathieu Blondel, Akinori Fujino, and Naonori Ueda. Large-scale multiclass support vector machine training via euclidean projection onto the simplex. In 2014 22nd International Conference on Pattern Recognition, pages 1289-1294. IEEE, 2014.
[34] Douglas P Hardin and Edward B Saff. Minimal riesz. energy point configurations for rectifiable d-dimensional manifolds. Advances in Mathematics, 1734)2214-- 2005.
[35] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
12

---

[36] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith. and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprini arXiv: /8/200009/ 2018.
[37] Seok-Ju Hahn, Minwoo Jeong, and Junghye Lee. Connecting low-loss subspace for personalized federated learning. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 505-515, 2022.
[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[39] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 245-255. leee, 2009.
[40] John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to begin? on the impact of pre-training and initialization in federated learning. In International Conference on Learning Representations, 2023.
[41] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint o 07360, 2016.
[42] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangehuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. In AAAI Conference on Artificial Intelligence, volume 35, pages 7865-7873, 2021.
[43] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pages 7252-7261, 2019.
[44] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Fedde: Federated learning with non-iid data via local drift decoupling and correction. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
[45] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pages 1321-1330, 2017.
[46] Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. In International Conference on Learning Representations, 2020.
[47] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021.
[48] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143, 2020.
[49] Bo Li, Mikkel Schmidt, Tommy Alstrom, and Sebastian U Stich. On the effectiveness of partial variance reduction in federated learning with heterogeneous data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3964-3973, 2023.
[50] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[51] Hong-Yon You Chen, Cheng-Hao Tu, Ziwei Li, Han Wei Shen, and Wei-Lun Chao. On the importance and applicability of pre-training for federated learning. In International Conference on Learning Representations, 2022.
13

---

[52] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1-46 07 2015.
[53] Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus-Robert Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3):44--278, 2021.
[54] Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus-Robers Müller, and Marina M-C Höhne. Dora: Exploring outlier representations in deep neural networks. Transactions on Machine Learning Research, 2023.
[55] A. Rodio, F. Faticanti, O. Marfoq, G. Neglia, and E. Leonardi. Federated learning under heterogeneous and correlated client availability. In IEEE International Conference on Computer Communications, 2023.
[56] Philipp Wiesner, Ramin Khalili, Dennis Grinwald, Pratik Agrawal, Lauritz Thamsen, and Odej Kao. Fedzero: Leveraging renewable excess energy in federated learning. In International Conference on Future and Sustainable Energy Systems. ACM, 2024.
[57] Richard Burden and Douglas Faires. 2.1 the bisection algorithm. Numerical analysis, 1985.
14

---

Appendix
This appendix provides a nomenclature, details to our optimization problem and experimental setup, as well as additional results and insights.
Table 5: Nomenclature.
Symhol
Description
Clients
Communication rounds
T
Local training iterations
Purticipating clients in round
Mini-
Client gradient descent step size
Dr
Truining data of client
N
Total number of samples
Number of sumples at client datu distribution of client
pe(a. v)
wh E GRER
Global model at round
w ERD
Model of client k at round 1
A
(a 10.1] ||||||
M-dimenssionl stundard simplex
0
-
Simplex endpoints at round
w..
Em Cr 8m
Model parameters point A
Subregion rudius
Ra
Assigned subregion of client k
TE [T....7]
Suhregion assignment
Na
E RM
dimensional representation stucked gradient update (A0%,
client
A Optimization Problem
The Lagrangian of the lower-level optimization problem in ( has the following formulation
C(as, A) 1 3|eex. k + A(1 ax z) with À E IR being the Langrange multiplier. The "1
Lagrangian be further rewritten L(ors A) = [l (AS A1)l X(1 KR 2) X2n such that the optimization problem reduces to solving
min sER
2 a
(KE x1)113
(14)
subject to: 0g x 0.
(15)
The optimal solution of (14) is given by ax = [K 1²1 Plugging it back into the Lagrangian get the following dual function
C(ork 2)
||AE x011+ (A& x1)11 A(1 Rs - 2) X²n
(16)
| 1-1]
>(1 Kk 2) 22n
(17)
Finding of can be achieved by maximizing 17 using for example the bisection algorithm [57 After that the projected points are obtained aj [K A-1]+
B Training Hyperparameters
Table 6 summarizes all hyperparameters that were used for each dataset/model combination. We train CifarCNN on CIFAR-I0 for a total of 500 communication rounds, R8sNet-18 on CIFAR-10 for 100 communication rounds, FemnistCNN on FEMNIST for 350 rounds, and SqueezeNetVi on FEMNIST for 1000 rounds. Moreover, we train each setting using a total of 100 clients, and for FEMNIST we select randomly chosen subset of 100 total clients for each trial, of which we select randomly to participate training each communication round, except for CifarCNN on CIFAR-10 where we select 30 out of 100 clients to participate in each round. We evaluate all clients after every ten communication rounds. For CIFAR-10 we train a CifarCNN with batch size 50 using SGD with learning rate of 0.02, momentum of 0.5, and weight decay of 10-5 and pre- pre-trained
15

---

ResNet-18 18 with learning rate of batch size 32, using SGD with learning rate of 0.01, momentum of 0.9, and weight decay of 10-4 -4 For FEMNIST we train a pre- trained SqueezeNet with batch size 32 using SGD with learning rate of 0.005, momentum of O, weight decay of 10-2, and a FemnistCNN with batch size 32, learning rate 0.1, momentum O, weight decay of O. For FedProx we set the proximity hyperparameter to μ 0.01 for all settings. For DITTO, FEDROD and FEDPER we set the local epochs to the same value as epochs for the global model, i.c. D = E. All training hyperparameters for CIFAR-I0 and FEMNIST on FemnistCNN were taken from [37] CIFAR-10
on a pre-trained ResNet-18 from 5 and FEMNIST on pre-trained SqueezeNet from 40
Table 6: Summary of used hyperparameters for training.
Dataset/Model
T
K
|S*|
E/
mom
wd
IIFAR-10/CifarNNT
500
100
30
50
0.02
0.5
10
0.01
LIFAR-IO/ReSNER
100
10
0.01
0.9
10-4
10
0.01
FEMNIST/FemmisINNN
350
100
10
32
0.1
0.0
0.0
0.01
FEMNISS//spueeeNNevi
1000
100
10
32
0.005
0.0
10
0.01
C Simplex Learning on all NN parameters
In Figure compare the global (left) and average client (right) test accuracy of FLOCO and FLOCO-AlI, where the latter applies simplex learning to all NN parameters. As expected, FLOCO-All converges to the same global and average local test accuracy, but needs more communication rounds to do so, since it needs to train more parameters.
Dir(0.5) Global test acc.
Dir(0.5) Avg. client test acc.
0.8
Test accuracy
Test accuracy
0.6
0.7
0.4
0.6
FLOCO-AII FLOCO
FLOCO-AII FLOCO
0.2
0.5
1000
1000
Communication round
Communication round
Figure 3: Global test accuracy.
Figure 4: Average local client test accuracy.
Figure 5: Comparing simplex learning on all network layers vs. only on the last fully-connected layer.
D Sensitivity to Parameter Setting
We investigate how stable the performance of FLOCO is for different hyperparameter settings. Specifically, tested FLOCO with the combination of 50, 100, 200 (subregion assignment time step) and p = 0.1, 0.2, 0.4 (radius of subregions). and show the average local client and global test accuracy for CifarCNN on CIFAR-10 5-Fold in Figure 6 We observe that the average local client test accuracy (left) increases for earlier subregion assignment starting points 7 and lower client subregion radiuses p. with the best reached test accuracy being approximately 4% better than the worst, i.e., 82.79% against 78. 18%. The intuition for this is that earlier client specialization in less overlapping regions allows for better personalization. On the other hand, as can be observed in the right heatmap of Figure 6 the global test performance is less sensitive to the choice of these hyperparameters,
16

---

Table Average global and local test accuracy on CIFAR-IO.
CIFAR-10
CifarCNN
pre-trained ResNet-18
5-Fold
Dir(0.3)
3-Fold
Dir(0.3)
SuPerFed
63.22
76.65
63.00
71.73
64.88
52.78
76.04
60.91
FLOCO
68.26
80.92
69.79
74.64
74.61
87.38
79.11
82.29
ie., 70.66% against 69.30%. This is because, after subregion assignment, the entire solution simplex remains to trained, making the midpoint (global model) of the simplex less sensitive to the specialization process for client distributions.
Avg. client test acc.
Global test acc.
0.82
E- 82.79%
82.33%
78.54%
69.54%
69.30%
69.76%
0.80
= 200
82 21%
82.01%
78.18%
70.66%
70.14%
70.35%
0.76
= 300
80.
70.47%
69.98%
69.83%
0.72
0.70
p= 0.02
p=0.1
p 0.5.5
0=0.02
p=
p=5.5
Figure 6: Local average client (left) and global (right) test accuracies for different subregion assign- ment time step т and subregion radius settings.
E
Comparing FLOCO to SuPerFed
We benchmark Ft.cOO against the SuPerFed baseline on the CIFAR-10, 5-Fold, as well as Dir(O.5) splits using both CifarCNN trained from scratch as well as pre-trained ResNet18, on both global well local test performance. As shown in Table 7 FLOco outperforms SuPerFed in all settings. Note, that for this benchmark we have implemented FLOCO as well as SUPERFED in the FL framework Flower 21
17

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our claims in the abstract and introduction are empirically proven and ex- plained our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
18

---

Justification: We show that, in some cases. our method does not exceed the performance of baseline method, however, we show that it saves up computational cost which is very relevant in the field. We discuss this point in more detail and give an alternative solution.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex- system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Our paper does not include any theoretic assumption or proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
19

---

Justification: Our paper gives detailed information on how to reproduce our results, including all hyperparameters, models and dataset splits needed as well as a detailed description for our algorithm. Moreover, we upload our code together with the submission which includes a README that documents how experiments can reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our code in the submission which includes a README with detailed description on how to run our method in order to reproduce the shown results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/gum www.:ddee//ooSSSbmii...c
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips http://cwhi/cceeeeeee////eaaaaaa
for more details.
20

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and test details, including models, data splits, hyperparameters, model architectures etc. that are necessary to reproduce our results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Fach of our experiments is run across 5 trial runs with different random seeds in order to show confidence intervals for training and test runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
21

---

Answer:
[No]
Justification:
We did not explicitly compute the resources needed.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and made sure to respect it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and funetioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
22

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release any data models that pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide citations for all used datasets, models, and baselines.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
There are no new assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
23

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "vHXvO7ZiyO", "reviewer": "Reviewer_bjGi", "summary": "The authors propose federated learning over connected modes (FLOCO), where clients\r are assigned local subregions in this simplex based on their gradient signals, and\r together learn the shared global solution simplex.", "questions": "1 Can the authors clarify the difference between the proposed method and other clustering federated learning algorithms?\n\n2 Can you specifically describe the implementation details of the algorithm? Or improve the description. For example, there is a lack of more detailed description for the generation of R.\n\n3 In the algorithm section, can the specific algorithm be explained more clearly?", "limitations": "The authors discuss the reasons for the poor experimental performance of the proposed methods", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is richer in the type of experiments.", "weaknesses": "The proposed methodology lacks innovation.\n\nThe experimental baseline is too old (2021) and there are many recent and similar clustered federated algorithms that lack comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply thankful for Reviewer bjGi for their thorough review. We are pleased to read that the reviewer appreciates the richness of our experiments, the soundness and presentation of our work. In the In the following, we answer the reviewer’s remarks and questions in more detail:\n\n***Reviewer’s remark:*** “The proposed methodology lacks innovation.”\n***Author’s answer:***   We argue that our paper has significant novelty and contributions: Our way of using solution simplexes for federated learning is unique; we employ a novel projection method that spreads FL clients well across the simplex;  and we show in many different experiments that FLOCO performs well for global AND personalized FL, which are usually problems tackled separately. Thus we would like to ask for clarification by the reviewer about why our method lacks innovation.\n\n***Reviewer’s remark:*** “The experimental baseline is too old (2021) and there are many recent and similar clustered federated algorithms that lack comparison.”\n***Author’s answer:*** We agree that the original baselines are old, and thus we report on new comparisons with 5 other global AND local FL baselines.  As shown in Figures 1 and 2 in the rebuttal pdf and Tables 1-4 in the global response, our FLOCO method outperforms all the tested baselines (the results were averaged over 5 different random seeds).  We implemented our FLOCO in the FL Bench [1] framework, and compared it with most of the baselines except SuPerFed, another state-of-the-art FL method that is not supported by FL Bench.  SuPerFed is implemented in our Flower-based framework, and compared with our original implementation of FLOCO.  Note that FLOCO run on the FL Bench framework and that run on our Flower-based framework do not exactly match because of the mismatch in the aggregation order of clients.  Due to the time limitation, we could have run experiments in the FL base framework only on CIFAR10 Dir(0.5) on a SimpleCNN.   For the camera-ready we will run these comparisons on all dataset split-model combinations.\n\n***Reviewer’s question 1:*** Can the authors clarify the difference between the proposed method and other clustering federated learning algorithms?\n***Author’s answer:***  Our method does not cluster anything but project clients onto the solution simplex, so that the degrees of freedom within the simplex is used to capture characteristics of each client.\n\n***Reviewer’s question 2:*** Can you specifically describe the implementation details of the algorithm? Or improve the description. For example, there is a lack of more detailed description for the generation of R.\n***Author’s answer:***\nR is simply a L1-ball around the projected point of each client.  For drawing samples from the uniform distribution over R, we perform the following:\nGenerate random points on the surface of the L1 unit ball by normalizing the absolute values of a randomly generated vector.\nScale these points to fit within the desired radius.\nTranslate the points by the given center.\nWe will add further algorithm details in Appendix. \n\n***Reviewer’s question 3:*** In the algorithm section, can the specific algorithm be explained more clearly?\n***Author’s answer:***  We will add further algorithm details in Appendix.  We are happy to clarify its algorithm details if the reviewer specifies on which algorithm our description is unclear, other than the generation of samples on R.\n\n***Reviewer’s limitation remark:*** The authors discuss the reasons for the poor experimental performance of the proposed methods\n***Author’s answer:***  We do not understand this comment.  We do not show poor performance of the proposed method, and thus do not discuss the reason.  Could you please elaborate more? \n\n[1] FL Bench: (https://github.com/KarhouTam/FL-bench)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jj6wJcuglJ", "reviewer": "Reviewer_44Bh", "summary": "A novel method is proposed to derive the mode connectivity over the simplex defined by the central server for an improved global model as well as local personalization performances in the federated settings.", "questions": "- Line 255: in Figure 2, the convergence speed of the global test accuracy & the average local test accuracy of the proposed method is slower than the baseline methods (when seeing round ~50), which is somewhat counterintuitive provided that the total gradient variance remains the smallest across all FL training rounds (which authors stated that it leads to the better performance, i.e., line 265-266). Could the authors please provide an explanation on this?\n- Line 255: same in Figure 2 (and similarly in Figure 3), the average local test accuracy is suddenly surged, starting from the 200th round. This is quite intriguing, and I also expect an explanation on this result.", "limitations": "The authors do not evaluate the proposed method out of vision classification task.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed objective using Riesz s-Energy regularization along with the Euclidean projection onto the simplex is novel and convincing. The paper is well structured and various analyses are provided with accessible illustrations. The hyperparameter details provided are helpful in reproducing the results.", "weaknesses": "- Line 25: is leverages -> leverages\n- Line 26, 33, 66, 129: weight parameter -> parameter (since parameters typically represents both weights and biases in NNs)\n- Line 60: over the clients -> for the $k$-th client\n- Line 96: (not sure) in the RHS, it may be $\\boldsymbol{w}_\\boldsymbol{\\alpha}$\n- Line 127: please consider adding equation numbers in each line of the pseudocode of Algorithm 1. \n- Line 148: missing citation\n- Lines 189-190: the baseline methods are not convincing. Since the proposed method explicitly induces the mode connectivity (i.e., line 25 per se) for the improved FL performance, it should have been compared to methods with similar motivations, e.g. [1], which was cited in the Related Works section (line 286) but not directly compared. Plus, authors should have also considered adding more personalization methods (e.g., [2], which shares the same strategy, i.e., only exchanging parameters of classification layer, for communication-efficient personalization in FL) as the authors aim to prove the effectiveness of `FLOCO` for improved personalization performances. \n- Line 212: please also report standard deviations on 5 different runs. \n- Lines 461-466: the communication rounds (100 and 500), the number of clients (100), and the number of per-round clients in CIFAR-10 (which is 30 out of 100) are somewhat deviated from the practical FL settings, which raises doubts about the scalability of the proposed method. It is more acceptable if the proposed method is specifically designed for the cross-silo FL settings, which usually assume a small number of stateful clients and (optionally) full participation across rounds. \n\n\n[1] Connecting low-loss subspace for personalized federated learning (2022)  \n[2] Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to Reviewer 44Bh for their detailed review. We are happy to read that the reviewer acknowledges the novelty of our method that exploits recent findings in mode connectivity to train a simplex in the FL setting, which improves local and global performance. Moreover, we appreciate that the reviewer found our proposed Riesz-energy-based Euclidean projection onto the unit simplex novel and convincing. Below, we will answer each of the reviewer’s weakness remark, question and limitation.\n\n***Reviewer’s remark:*** Reviewer points out typos, missing citations, naming conventions, e.g. weight parameter -> parameter, and style changes, e.g. inclusion of equations numbers.\n***Author’s answer:*** We have fixed all pointed out remarks above.\n\n***Reviewer’s remark:*** Lines 189-190: the baseline methods are not convincing. Since the proposed method explicitly induces the mode connectivity (i.e., line 25 per se) for the improved FL performance, it should have been compared to methods with similar motivations, e.g. [1], which was cited in the Related Works section (line 286) but not directly compared. Plus, authors should have also considered adding more personalization methods (e.g., [2], which shares the same strategy, i.e., only exchanging parameters of classification layer, for communication-efficient personalization in FL) as the authors aim to prove the effectiveness of FLOCO for improved personalization performances.\n***Author’s answer:*** We agree that the originally chosen baselines are not sufficient, and thus we report on new comparisons with 5 other global AND local FL baselines.  As shown in Figures 1 and 2 in the rebuttal pdf and Tables 1-4 in the global response, our FLOCO method outperforms all the tested baselines (the results were averaged over 5 different random seeds).  We implemented our FLOCO in the FL Bench [1] framework, and compared it with most of the baselines except SuPerFed, another state-of-the-art FL method that is not supported by FL Bench.  SuPerFed is implemented in our Flower-based framework, and compared with our original implementation of FLOCO.  Note that FLOCO run on the FL Bench framework and that run on our Flower-based framework do not exactly match because of the mismatch in the aggregation order of clients.  Due to the time limitation, we could have run experiments in the FL base framework only on CIFAR10 Dir(0.5) on a SimpleCNN.   For the camera-ready we will run these comparisons on all dataset split-model combinations.\n\n***Reviewer’s remark:*** Line 212: please also report standard deviations on 5 different runs.\n***Author’s answer:*** In Table 1-4 in the global response, we report on standard deviations over the 5 different seeded runs for each experiment.\n\n***Reviewer’s remark:*** Lines 461-466: the communication rounds (100 and 500), the number of clients (100), and the number of per-round clients in CIFAR-10 (which is 30 out of 100) are somewhat deviated from the practical FL settings, which raises doubts about the scalability of the proposed method. It is more acceptable if the proposed method is specifically designed for the cross-silo FL settings, which usually assume a small number of stateful clients and (optionally) full participation across rounds.\n***Author’s answer:*** We have tested our method on the FEMNIST dataset, a practical FL data set where different clients have different numbers of data points.\n\n***Reviewer’s question 1:*** Line 255: in Figure 2, the convergence speed of the global test accuracy & the average local test accuracy of the proposed method is slower than the baseline methods (when seeing round ~50), which is somewhat counterintuitive provided that the total gradient variance remains the smallest across all FL training rounds (which authors stated that it leads to the better performance, i.e., line 265-266). Could the authors please provide an explanation on this?\n***Author’s answer:*** The slow convergence of FLOCO in the beginning comes from the larger degrees of freedom (we train M end-points thus have M times more parameters to be trained). After the 200th round where we project the clients onto the solution simplex and thus the variance of gradient is significantly reduced, the global training is accelerated, as seen in Fig.2 (left) in the original submission (not as drastic as the local test accuracy, but still a larger slope than FedAve and DITTO can be observed).\n\n***Reviewer’s question 2:*** Line 255: same in Figure 2 (and similarly in Figure 3), the average local test accuracy is suddenly surged, starting from the 200th round. This is quite intriguing, and I also expect an explanation on this result.\n***Author’s answer:***  As mentioned in the caption of Fig. 2 in the submission, FLOCO performs the simplex learning (with alpha drawn from uniform distribution over the whole standard simplex) until the 200th round.  At the 200th round, FLOCO performs the proposed projection of the clients onto the simplex, based on the update signals, and starts using the solution simplex for personalization.  We will make this point clearer in the main text.\n\n***Reviewer’s limitation remark:*** The authors do not evaluate the proposed method out of vision classification task.\n***Author’s answer:*** We will extend our experimental setup to the google speech dataset for the camera-ready version.\n\n[1] Connecting low-loss subspace for personalized federated learning (2022)\n[2] Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020)\n[3] FL Bench: (https://github.com/KarhouTam/FL-bench)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bKFn2Wvn1n", "reviewer": "Reviewer_5sib", "summary": "In this study, the authors tackle challenges in federated learning by introducing FLOCO, which uses linear mode connectivity to identify a solution simplex in neural network weight space. This approach allows for personalized client model training within the simplex, while also enabling efficient updates to both global and local models.", "questions": "$\\textbf{Q1}.$  Choosing an appropriate experimental setup is often overlooked in the pFL setting. Note that the following two scenarios are inappropriate for pFL experiments: $\\textbf{(1)}$ Clients have enough data to train their local models independently, eliminating the need to participate in an FL system. $\\textbf{(2)}$ The data distribution of the clients is identical or nearly identical. This typically results in high accuracy for FedAvg, making it challenging to justify using pFL methods. It would be informative to evaluate your method in experimental setups where local training and FedAvg fail to provide good results and compare your method's performance with SOTA  pFL methods, especially methods with decoupled parameter space such as FedPer [1] and FedRep [2] and similarity-based methods such as pFedSim [3].\n\n$\\textbf{Q2}.$ As previously mentioned, evaluating the performance of local models on their respective local datasets is crucial in personalized federated learning (pFL). Including the performance metrics of these models (average performance when trained solely on local training data) would provide valuable insights into the experimental results.\n\n[1] Arivazhagan, Manoj Ghuhan, et al. \"Federated learning with personalization layers.\" arXiv preprint arXiv:1912.00818 (2019).\n\n[2] Collins, Liam, et al. \"Exploiting shared representations for personalized federated learning.\" International conference on machine learning. PMLR, 2021.\n\n[3] Chen, Yizhu, et al. \"PFedSim: An Efficient Federated Control Method for Clustered Training.\" IEEE Journal of Radio Frequency Identification 6 (2022): 779-782.", "limitations": "The discussion of the limitations of the proposed method is framed within the context of future directions. It is suggested to address these limitations separately, focusing on theoretical analysis and validation using real-world federated learning datasets suited for personalized FL (pFL).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces an interesting application of the linear mode connectivity approach in a personalized FL (pFL) setting. This method can be considered an alternative to distance-based and similarity-based pFL algorithms. The algorithm is evaluated under various experimental setups and compared with standard and personalized FL algorithms.", "weaknesses": "To strengthen their empirical findings, the authors could benefit from leveraging the bilevel optimization literature for analyzing the convergence of their algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very pleased and thankful for Reviewer 5sib for their thorough review. In particular we are thankful that the reviewer found our application of linear mode connectivity to the FL setting interesting and the soundness, presentation, contribution and experimental setup of our paper good.\nIn the following, we will answer the reviewer’s remarks and questions in detail:\n\n***Reviewer’s remark:*** “To strengthen their empirical findings, the authors could benefit from leveraging the bilevel optimization literature for analyzing the convergence of their algorithm.”\n***Author’s answer:*** Thank you for your suggestion.  We will consider theoretical analysis for future work.\n\n***Reviewer’s question 1:*** “Choosing an appropriate experimental setup is often overlooked in the pFL setting. Note that the following two scenarios are inappropriate for pFL experiments: (1) Clients have enough data to train their local models independently, eliminating the need to participate in an FL system. (2) The data distribution of the clients is identical or nearly identical. This typically results in high accuracy for FedAvg, making it challenging to justify using pFL methods. It would be informative to evaluate your method in experimental setups where local training and FedAvg fail to provide good results and compare your method's performance with SOTA pFL methods, especially methods with decoupled parameter space such as FedPer [1] and FedRep [2] and similarity-based methods such as pFedSim [3].”\n***Author’s answer:*** We agree that the originally chosen baselines are not sufficient, and thus we report on new comparisons with 5 other global AND local FL baselines.  As shown in Figures 1 and 2 in the rebuttal pdf and Tables 1-4 in the global response, our FLOCO method outperforms all the tested baselines (the results were averaged over 5 different random seeds).  We implemented our FLOCO in the FL Bench [1] framework, and compared it with most of the baselines except SuPerFed, another state-of-the-art FL method that is not supported by FL Bench.  SuPerFed is implemented in our Flower-based framework, and compared with our original implementation of FLOCO.  Note that FLOCO run on the FL Bench framework and that run on our Flower-based framework do not exactly match because of the mismatch in the aggregation order of clients.  Due to the time limitation, we could have run experiments in the FL base framework only on CIFAR10 Dir(0.5) on a SimpleCNN.   For the camera-ready we will run these comparisons on all dataset split-model combinations.\n\n***Reviewer’s question 2:*** As previously mentioned, evaluating the performance of local models on their respective local datasets is crucial in personalized federated learning (pFL). Including the performance metrics of these models (average performance when trained solely on local training data) would provide valuable insights into the experimental results.\n***Author’s answer:*** Since we have been focusing on additional comparisons with many state-of-the-art baselines, we could not have made it for comparison with this basic baseline.  We expect that the local training is at least much slower than the federated learning for the clients that only get a small number of samples.  We will conduct this basic experiment in the camera ready.\n\n***Reviewer’s limitation remark:*** The discussion of the limitations of the proposed method is framed within the context of future directions. It is suggested to address these limitations separately, focusing on theoretical analysis and validation using real-world federated learning datasets suited for personalized FL (pFL).\n***Author’s answer:*** We will follow your suggestion.\n\n[1] FL Bench: (https://github.com/KarhouTam/FL-bench)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2wG71M6Ez", "reviewer": "Reviewer_a16f", "summary": "This paper proposes a method for FL to train a simplex of linearly connected solutions with uniformly low loss. Clients are expressed in the simplex by projecting the gradient onto the simplex and similar clients are close to each other. In each communication round each client samples points in the neighborhood of their projected point and jointly train the solution simplex.\n\nEach client is expressed as a point in the simplex based on the gradient update signals. The subregion assignments preserve the similarity of the clients by applying the Euclidean projection onto the positive simples with Riesz s-Energy regularization - this spreads the projections while maintaining the similarity between the clients. As the gradient update signals are informative for the subregion assignment only after the model has been trained for some time, the first stage of the algorithm is FedAvg with simplex learning.", "questions": "- Could you clarify how simplex learning works in practice? Do you compute the loss and the gradients on each of the endpoints or do you sample one point in the simplex and backpropagate to update the endpoints? On which weights do you compute the loss? I think this is unclear from the paper and it would be beneficial to add a section “Simplex Learning” for the reader.\n- From my understanding, the idea of using a shared simplex and then dividing it into subregions is to have both global and local classifiers and to incentivize knowledge transfer and robustness. I would also see two other baselines/ablations that the authors might want to try to corroborate that this is actually beneficial. First, fedavg simplex learning (no subregions, check global accuracy) and then local fine-tuning for personalization (compute local accuracy) personalization can be done with local simplex learning or standard learning. I guess this could be a way to study the impact of simplex learning by itself and the importance of subregions.\n- As the authors mentioned in the related work, there are other personalized FL methods that apply simplex learning. Why did the authors not report their results as baseline? E.g. references 29 and 19.\n- I’m not sure I fully buy the claims and observations in the paragraph on variance reduction (lines 255-272). The main issue with heterogeneity is convergence speed, so to claim some effect on this the authors should show faster convergence on the global accuracy. There is evidence that the effect of the subregions impacts local accuracy, but this is expected as there is better specialization - as it would be with a local fine-tuning stage.\n- I suggest the authors also have a look at other more recent methods for global learning such as scaffold, feddyn, mime that could help with heterogeneity. They could also be used in conjunction with simplex learning.\n- I found it odd that the authors decided to show the plots in Figure 2 using the simpleCNN and FEMNIST while from table 1 the cifar10 experiments show more edge of FLOCO over the baselines.\n- Basically, the method only applies to the last FC layer - did the authors try it on the full model?", "limitations": "Limitations are not clearly expanded and should be discussed in a section of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the proposed approach is novel and interesting. The method applies simplex learning to tackle personalized federated learning.", "weaknesses": "Some details of the algorithm should be better explained. Some baselines seem to be missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very pleased and thankful for Reviewer a16f for their thorough review. In particular we are thankful that the reviewer found our method novel and interesting, and the soundness, presentation, contribution and experimental setup of our paper good.\nIn the following, we will answer the reviewer’s remarks and questions in more detail:\n\n***Reviewer’s remark:*** “Some details of the algorithm should be better explained. Some baselines seem to be missing.”\n***Author’s answer:*** We will explain the algorithm in more detail in the camera ready version. Which parts should be explained better in particular?  We report on new comparisons with 5 other global AND local FL baselines.  As shown in Figures 1 and 2 in the rebuttal pdf and Tables 1-4 in the global response, our FLOCO method outperforms all the tested baselines (the results were averaged over 5 different random seeds).  We implemented our FLOCO in the FL Bench [1] framework, and compared it with most of the baselines except SuPerFed, another state-of-the-art FL method that is not supported by FL Bench.  SuPerFed is implemented in our Flower-based framework, and compared with our original implementation of FLOCO.  Note that FLOCO run on the FL Bench framework and that run on our Flower-based framework do not exactly match because of the mismatch in the aggregation order of clients.  Due to the time limitation, we could have run experiments in the FL base framework only on CIFAR10 Dir(0.5) on a SimpleCNN.   For the camera-ready we will run these comparisons on all dataset split-model combinations.\n\n***Reviewer’s question 1:*** Could you clarify how simplex learning works in practice? Do you compute the loss and the gradients on each of the endpoints or do you sample one point in the simplex and backpropagate to update the endpoints? On which weights do you compute the loss? I think this is unclear from the paper and it would be beneficial to add a section “Simplex Learning” for the reader.\n***Author’s answer:*** We sample one point in the simplex and backpropagate to update the endpoints.  The loss to be minimized in the simplex learning (where alpha is drawn from the uniform distribution) is given in Eq.(7) in the original submission.  Since FLOCO applies simplex learning only to the last layer, the weights in the other layers are point-estimated, which amounts to tying all end-points to a single point.  We will extend Section 2.2 with these details. \n\n***Reviewer’s question 2:*** From my understanding, the idea of using a shared simplex and then dividing it into subregions is to have both global and local classifiers and to incentivize knowledge transfer and robustness. I would also see two other baselines/ablations that the authors might want to try to corroborate that this is actually beneficial. First, FedAvg simplex learning (no subregions, check global accuracy) and then local fine-tuning for personalization (compute local accuracy) personalization can be done with local simplex learning or standard learning. I guess this could be a way to study the impact of simplex learning by itself and the importance of subregions.\n***Author’s answer:*** Thank you for your suggestion.  We show these results in Figs. 3 and 4 in the rebuttal pdf, and will show them in the camera ready.\n\n***Reviewer’s question 3:*** As the authors mentioned in the related work, there are other personalized FL methods that apply simplex learning. Why did the authors not report their results as baseline? E.g. references 29 and 19.\n***Author’s answer:*** The other simplex learning approach for federated learning, i.e. SuPerFed[1], is now included in Table 2 in the global response, which shows that FLOCO outperforms SuPerFed.  Moreover, we have benchmarked FLOCO against many other baselines for the SimpleCNN - Dit(0.5) setting and the results are shown in Table 1, where our method outperforms the other methods. For the camera-ready version, we will show complete comparisons on all dataset-model combinations\n\n***Reviewer’s question 4:*** I’m not sure I fully buy the claims and observations in the paragraph on variance reduction (lines 255-272). The main issue with heterogeneity is convergence speed, so to claim some effect on this the authors should show faster convergence on the global accuracy. There is evidence that the effect of the subregions impacts local accuracy, but this is expected as there is better specialization - as it would be with a local fine-tuning stage.\n***Author’s answer:*** FLOCO has M times more parameters to be trained because it learns the end-points of the solution simplex.  For this reason, the simplex learning is slower than the baselines until the 200th round, where FLOCO projects the clients onto the solution simplex and thus the variance of gradients gets significantly reduced.  After the 200th round, the consistent gradient signals of FLOCO accelerate the training and thus FLOCO converges to the model with higher global test accuracy.  We can observe  in Fig.2 (left) in the original submission steeper slopes in the global test accuracy after the 200th round, compared to FedAve and DITTO, although the difference is not as drastic as in the local accuracy.\n\n***Reviewer’s question 5:*** I suggest the authors also have a look at other more recent methods for global learning such as scaffold, feddyn, mime that could help with heterogeneity. They could also be used in conjunction with simplex learning.\n***Author’s answer:*** We have been focusing on comparison with many state-of-the-art baselines in the rebuttal, and unfortunately did not have time to evaluate the suggested baselines.\n\n***Reviewer’s question 6:*** I found it odd that the authors decided to show the plots in Figure 2 using the simpleCNN and FEMNIST while from table 1 the cifar10 experiments show more edge of FLOCO over the baselines.\n***Author’s answer:*** We chose simmpleCNN-FEMNIST because it shows a typical behavior.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their detailed review, including remarks, questions and suggestions for improvement. As most reviewer’s pointed out, some important baselines, such as SuPerFed[1] and others are crucial to benchmark against our method. We have thus extended our experimental setting and have included 5 more baselines. As our new results suggest, FLOCO outperforms all benchmarked methods including the state-of-the-art. 
The main experimental results that we have computed during the rebuttal phase are listed in the 4 tables below. We reference them throughout our whole rebuttal. 
Tables 1 & 2 show the benchmark of FLOCO and another method that employs mode connectivity which is called SuPerFed, for global and average local test performance respectively.
Tables 3 & 4 show the benchmark of FLOCO against 5 baselines from the FL-Bench repository, for global and local test performance respectively.
In the rebuttal pdf, in Figures 1 & 2 we have included the training curves for the latter. Moreover, we have included a comparison plot of FedAvg with simplex learning compared to Floco, as suggested by Reviewer a16f in Figures 3 & 4.
Lastly, Figures 5 & 6 in the rebuttal pdf show how Floco performs when training a simplex over each parameter in each layer. As expected, due to the higher degrees of freedom, Floco over all parameters, Floco-All, needs more time to converge to the same performance as Floco with simplex learning only on the last layer.


[1] Connecting low-loss subspace for personalized federated learning (2022)

[2] FL Bench: (https://github.com/KarhouTam/FL-bench)

# Table 1: Floco against SuPerFed - Global test accuracy:

| Model-Dataset-Split                    | FLOCO | SuPerFed |
|----------------------------------------|-------|----------|
| SimpleCNN - CIFAR10 - 5-Fold           | 68.26 | 63.22    |
| SimpleCNN - CIFAR10 - Dir(0.5)         | 69.79 | 63.00    |
| SimpleCNN - FEMNIST - -                | 77.95 | 76.80    |
| PretrainedResnet18 - CIFAR10 - 5-Fold  | 74.61 | 64.88    |
| PretrainedResnet18 - CIFAR10 - Dir(0.5)| 79.11 | 76.04    |

# Table 2: Floco against SuPerFed - Avg. local test accuracy:

| Model-Dataset-Split                    | FLOCO | SuPerFed |
|----------------------------------------|-------|----------|
| SimpleCNN - CIFAR10 - 5-Fold           | 80.92 | 75.65    |
| SimpleCNN - CIFAR10 - Dir(0.5)         | 74.64 | 71.73    |
| SimpleCNN - FEMNIST - -                | 83.89 | 82.59    |
| PretrainedResnet18 - CIFAR10 - 5-Fold  | 87.38 | 52.78    |
| PretrainedResnet18 - CIFAR10 - Dir(0.5)| 82.29 | 60.91    |

# Table 3: Floco against 5 baselines - Global test accuracy

| Model-Dataset-Split                    | FLOCO | Ditto | FedAvg | FedRep | FedPac | FedDyn |
|----------------------------------------|-------|-------|--------|--------|--------|--------|
| SimpleCNN - CIFAR10 - Dir(0.5)         | 73.44 | 73.06 | 72.94  | 43.70  | 68.70  | 31.67  |

# Table 4: Floco against 5 baselines - Avg. local test accuracy

| Model-Dataset-Split                    | FLOCO | Ditto | FedAvg | FedRep | FedPac | FedDyn |
|----------------------------------------|-------|-------|--------|--------|--------|--------|
| SimpleCNN - CIFAR10 - Dir(0.5)         | 79.54 | 79.20 | 73.42  | 68.00  | 73.15  | 31.39  |






# Extended answer to reviewer a16f:
***Reviewer’s question 7:*** Basically, the method only applies to the last FC layer - did the authors try it on the full model?
***Author’s answer:*** We show the result with FLOCO-all in Figures 5 and 6 in the rebuttal pdf, where we observe that applying FLOCO to all layers significantly reduces training speed.

Meta-review: **Summary:**

FLOCO proposes using linear mode connectivity to identify a solution simplex in neural network weight space for federated learning. Clients are assigned subregions in the simplex based on gradient signals to allow personalization while jointly learning a global solution. Claims to accelerate global training and improve local accuracy compared to baselines.

**Strengths:**

- Novel application of linear mode connectivity/simplex learning to federated learning
- Well-written paper with comprehensive experiments
- Addresses both global and personalized FL in a unified framework
- Achieves good performance on both global and local accuracy metrics

**Weaknesses:**

- Limited theoretical analysis
- Some baseline comparisons missing initially (addressed in rebuttal)
- Scalability to very large numbers of clients unclear
- Applicability may be limited to cross-silo rather than cross-device FL settings

**Key Discussion Points:**

- *Baseline comparisons:* Authors added several new baselines in rebuttal, showing FLOCO outperforms them
- *Applicability:* Authors acknowledged FLOCO is more suited for cross-silo FL with stateful clients, not cross-device scenarios
- *Scalability:* Questions raised about performance with thousands of clients; authors agreed this is a limitation for low-data clients
- *Ablation studies:* Additional experiments requested and provided to isolate effects of different components
- *Theoretical analysis:* Lack of convergence analysis noted; authors agreed this is future work
- *Implementation details:* Clarifications provided on simplex learning process

Overall, reviewers generally found the core idea interesting and results promising, but had some concerns about scope/applicability and analysis depth. Scores ranged from borderline reject to weak accept, with most leaning positive after author responses. The consensus seems to be that this is a solid contribution to federated learning research, particularly for cross-silo scenarios, though some limitations remain.
Decision: Accept (poster)