Federated Learning over Connected Modes

Authors: dennis  grinwald
philipp  wiesner
shinichi  nakajima
  bifold
t u berlin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:45:30

Abstract: Statistical heterogeneity in federated learning poses two major challenges: slow global training due to conflicting gradient signals, and the need of personalization for local distributions. In this work, we tackle both challenges by leveraging recent advances in \emph{linear mode connectivity} --- identifying a linearly connected low-loss region in the parameter space of neural networks, which we call solution simplex. We propose federated learning over connected modes (\textsc{Floco}), where clients are assigned local subregions in this simplex based on their gradient signals, and together learn the shared global solution simplex. This allows personalization of the client models to fit their local distributions within the degrees of freedom in the solution simplex and homogenizes the update signals for the global simplex training.  Our experiments show that \textsc{Floco} accelerates the global training process, and significantly improves the local accuracy with minimal computational overhead in cross-silo federated learning settings.

Main paper: 
 introduction: 
 Federated learning (FL) [1] is a decentralized machine learning paradigm that facilitates collaborative model training across distributed devices while preserving data privacy. However, in typical real applications, statistical heterogeneity-non-identically and independently distributed (non-IID) data distributions at clients-makes it difficult to train well-performing models. To tackle this difficulty, various methods have been proposed, e.g., personalized FL [2] , clustered FL [3] , advanced client selection strategies [4] , robust aggregation [5] , and federated meta-and multi-task learning approaches [6] . These methods aim either at training a global model that performs well on the global distribution [7] , or, as it is common in personalized FL, at training multiple client-dependent models each of which performs well on its local distribution [8] . These two aims often pose a trade-off-a model that shows better local performance tends to suffer from worse global performance, and vice versa. In this work, we aim to develop a FL method that improves local performance compared to state-of-the art methods without sacrificing global performance.Our approach leverages recent findings on mode connectivity [9] [10] [11] -the existence of low-loss paths in the parameter space between independently trained neural networks-and its applications [12] . These works show that minima for the same task are typically connected by simple low-loss curves, and that this connectivity benefits training for multi-task and continual learning. In particular, the authors show that embracing mode connectivity between models improves accuracy on each task and remedies the risk of catastrophic forgetting.In this paper, we leverage such effects, and propose federated learning over connected modes (FLOCO) , where the clients share and together train a solution simplex-a linearly connected low-loss region in the parameter space. Specifically, FLOCO represents clients as points within the standard simplex based on the similarity between their gradients, and assigns each client a specific subregion of the simplex. Clients then participate in FL by sampling different models within their assigned subregions and sending back the gradient information to update the vertices of the global solution simplex (see Figure 1 : FLOCO expresses each client as a point (⋆ in the top-center plot) by projecting the gradient signals onto the simplex, so that similar clients are close to each other. In each communication round, each client uniformly samples points in the neighborhood of their projected point (top-right plot), and jointly train the solution simplex. The lower row shows the resulting test loss on the solution simplex, where the loss for the global distribution (left) is uniformly small, while the losses for individual local distributions (center for client 1 and right for client 2) are small around their projected points. Fig. 1 ). This method facilitates collaborative training through the common solution simplex, while allowing for client-specific personalization according to their local data distributions.Our experiments show that FLOCO outperforms common FL baselines (FedAvg [1] , FedProx [13] ) and state-of-the-art personalized FL approaches (FedRoD [14] , APFL [15] , Ditto [16] , FedPer [17] ) on both local and global test metrics-without introducing significant computational overhead-in cross-silo FL settings. We also demonstrate additional benefits of FLOCO, including better uncertainty estimation, improved worst client performance, and smaller divergence of gradient signals.Our main contributions are summarized as follows:• We propose FLOCO, a novel FL method that trains a solution simplex for mitigating the statistical heterogeneity of clients, and demonstrate its state-of-the-art performance for local personalized FL.• We propose a simple projection method to express clients as points in the standard simplex based on the gradient signals, and establish a procedure of subregion assignments.• We conduct experimental evaluations on semi-artificial and real-world FL benchmarks with detailed analyses of the behavior of FLOCO, which give insights into how the mechanism improves performance compared to the baselines.We provide implementations of FLOCO in the FL frameworks FL-bench [18] and Flower [19] . Our code is publicly available: https://github.com/dennis-grinwald/floco. 
background: 
 In this section, we briefly explain the concepts behind federated learning and mode connectivity, which form the backbone of our approach. The symbols that we use throughout the paper are listed in Table 5 in Appendix.Assume a federated system where the server has a global model g 0 and the K clients have their local models {g k } K k=1 . FL aims to obtain the best performing models {g * k } K k=0 such thatEQUATIONEQUATIONwhereF * k (g) = E (x,y)∼p k (x,y) [f (g, (x, y))] .Here, p(k) is the normalized population of data samples for the k-th client, p k (x, y) is the data distribution for the client k, and f (g, (x, y)) is the loss, e.g., cross-entropy, of the model g on a sample (x, y) ∈ R I × {1, . . . , L}, where I is the dimension of an input data sample. Global [20] and personalized [8] FL aim to approximate g * 0 and {g * k } K k=1 , respectively, by using the training data D = {D k } K k=1 observed by the clients. Throughout the paper, we assume that all models are neural networks (NNs) y = g k (x; w k ) with the same architecture, and represent the model g k with its NN parameters w k ∈ R D , i.e., we hereafter represent g k (x; w k ) by w k and thus denote, e.g.,F * k (g k ) by F * k (w k ). Let N = K k=1N k be the total number of samples, where N k = |D k |. For the independent and identically distributed (IID) data setting, i.e., p k (x, y) = p(x, y), ∀k = 1, . . . , K, the global and personalized FL aim for the same goal, and the minimum loss solution for the given training data isEQUATIONwhereF k (w) = 1 N k (x,y)∈D k f (w, (x, y)).In this setting, Federated Averaging (FedAvg) [1] ,EQUATIONis known to converge to w 0 , and thus solve Eq. ( 3). Here, S t is the set of clients that participate the t-th communication round, and ∆w t+1 k = w t+1 k -w t 0 is the update after T ′ steps of the local gradient descent,EQUATIONwhere w0 = w t 0 , wT ′ = w t+1 k , and γ is the step size. FedAvg has been further enhanced with, e.g., proximity regularization [21] , auxiliary data [22] , and ensembling [23] .On the other hand, in the more realistic non-IID setting, where w * 0 ̸ = w * k , FedAvg and its variants suffer from slow convergence and poor local performance [24] . To address such challenges, Ditto [16] was proposed for personalized FL, i.e., to approximate the best local models {w * k } K k=1 . Ditto has two training phases: it first trains the global model w 0 by FedAvg, then trains the local models with proximity regularization to w 0 , i.e.,w k = argmin w k F k (w k , w 0 ) ≡ F k (w k ) + λ 2 ∥w k -w 0 ∥ 2 2 ,where λ controls the divergence from the global model. Ditto has been shown to outperform many other non-IID FL methods, including the client clustering method HYPCLUSTER, adaptive federated learning (APFL), which interpolates between a global and local models [25] , Loopless Local SGD (L2SGD), which applies global and local model average regularization [26] , and MOCHA [6] , which fits task-specific models through a multi-task objective.Freeman and Bruna (2017) [27] , as well as Garipov et al. (2018) [10] , discovered the mode connectivity in the NN parameter space-the existence of simple regions with low training loss between two well-trained models from different initializations. Nagarajan and Kolter (2019) [11] showed that the path is linear when the models are trained from the same initialization, but with different ordering of training data. Frankle et al. (2020) [28] showed that the same pre-trained models stay linearly-connected after fine-tuning with gradient noise or different data ordering. [29] found that the low loss connection is not necessarily in 1D, and [30] showed that a simplex,EQUATIONwithin which any point has a small loss, can be trained from randomly initialized endpoints.Here, {θ m ∈ R D } M +1 m=1 are the endpoints or vertices of the simplex, and ∆ M = {α ∈ [0, 1] M +1 ; ∥α∥ 1 = 1} denotes the M -dimensional standard simplex. This simplex learning is performed by finding the endpoints that (approximately) minimizeEQUATIONwhere U W denotes the uniform distribution on a set W. During training, one model realization w α from the simplex gets sampled and its gradient update wrt. the loss, e.g. cross-entropy, gets backpropagated to the simplex endpoints {θ m } M +1 m=1 . 
proposed method: 
 In this section, we introduce our approach, where the mode connectivity is leveraged for collaborative training between personalized client models.The main idea behind FLOCO is to assign subregions of the solution simplex (6) to clients in such a way that similar clients train neighboring (and overlapped) regions, while enforcing (linear) connectivity to all other client's subregions. The connectivity constraint systematically regularizes client training and allows for efficient collaboration between them.The subregion assignments need to reflect the similarity between the clients. To this end, FLOCO expresses each client as a point in the standard simplex, based on the gradient update signals.Specifically, it applies the Euclidean projection onto the positive simplex [31] with the Riesz s-Energy regularization [32] , which gives well spreaded projections that preserve the similarity between the client's gradient signals as much as possible. Once the clients are projected onto the standard simplex as {α k ∈ ∆ M } K k=1 , we assign the L1-ball with radius ρ around α k , i.e., R k = {α ∈ ∆ M ; ∥α -α k ∥ 1 ≤ ρ}, to the k-th client. Note that the gradient update signals are informative for the subregion assignment only after the (global) model is trained to some extent. Therefore, the subregion assignment is performed after τ FL rounds are performed. Before the assignment, i.e., t ≤ τ , all clients train the whole standard simplex R k = ∆ M , ∀k, which corresponds to a simplex learning version of FedAvg.Starting from randomly initialized simplex endpoints {θ m } M +1 m=1 , FLOCO performs the following steps for each participating client k ∈ S t in each communication round t:1. The server sends the current endpoints {θ t m } M +1 m=1 to the client k. 2. The client k performs simplex learning only on the assigned subregion R k as a local update.3. The client sends the local update of the endpoints to the server. This way, FLOCO is expected to learn the global solution simplex {w α ; α ∈ ∆ M }, while allowing personalization to local client distributions within the solution simplex. Algorithm 1 shows the main steps.Although the simplex learning can be applied to all parameters, our preliminary experiment showed that applying simplex learning only of the parameters in the last fully-connected layer (while pointestimating the other parameters) is sufficient. Therefore, our FLOCO only applies the simplex learning to the last layer, which gives other benefits including applicability to fine-tuning of pre-trained models, and significant reduction of computational and communication costs, as shown in Section 4.3.Below, we describe detailed procedures of client projection, local and global updates in the communication rounds, and inference in the test time.Algorithm 1: Federated Learning over Connected Modes (FLOCO).:number of communication rounds T , number of clients K, simplex dimension M , subregion assignment round τ , subregion radius ρ{θ 0 m } M +1 m=1 ← initialize_simplex(M ) R k ← ∆ M , ∀k = 1, . . . , K //set all client subregions to the whole standard simplex for t = 1 to T doif t = τ then {{∆θ τ m,k } M +1 m=1 } K k=1 ← collect_and_stack_gradients() {α k } K k=1 ← client_representation({{∆θ τ m,k } M +1 m=1 } K k=1 ) {R k } K k=1 ← assign_subregions({α k } K k=1 , ρ) S t ← choose_participating_clients() for k ∈ S t do {θ t+1 m,k } M +1 m=1 ← local_update({θ t m,k } M +1 m=1 , R k ) {θ t+1 m } M +1 m=1 ← global_update({{θ t+1 m,k } M +1 m=1 } k∈S t )We explain how to obtain the representations {α k ∈ ∆ M } of the clients in the standard simplex such that similar clients are located close to each other, while all clients are well-spread across the simplex.At communication round t = τ , FLOCO uses the gradient updates of the endpoints {∆θ τ m,k } M +1 m=1 as a representation of the client k. We concatenate the gradients for the M + 1 endpoints into a ((M + 1) • D)-dimensional vector, and apply the PCA projection onto the M dimensional space, yielding κ k ∈ R M as a low dimensional representation. To project {κ k } onto the standard simplex ∆ M , we solve the following minimization problem:EQUATIONsubject to:EQUATIONThe objective function in Eq. ( 8) is the Riesz s-Energy [32] , a generalization of potential energy of multiple particles in a physical space, and therefore its minimizer correponds to the state where particles are well spread across the space. The minimization in the constraint (9) corresponds to the Euclidean projection onto the positive simplex [31] , which forces {β k } to keep the locations of the PCA projections {κ k } of the clients. Fortunately, this minimization problem (for a fixed z) is convex, and can be efficiently solved (see Appendix A). We solve the main problem (8) by computing β k (z) on a 1D grid in z ∈ [0, 1] with the interval 0.001, and set the representations of the clients toα k = β k ( z)z , where z is the minimizer of Eq. ( 8).In the t-th communication round, the server sends the current endpoints {θ t m } M +1 m=1 to the participating clients S t . Then, each client k ∈ S t draws one sample per mini-batch from the uniform distribution A = {α b } B b=1 ∼ U R k on the assigned subregion and applies T ′ local updates,EQUATIONto the endpoints with α sequentially chosen from A. 1 Hereθ0 m = θ t m , θT ′ m = θ t+1 m,k . The local updates {∆θ t+1 m,k = θ t+1 m,k -θ t m } M +1m=1 are sent back to the server, which updates the endpoints asEQUATIONAs explained in Section 3.1, the client subregions are initially set to the whole simplex ∆ M before the subregion assignment is performed at t = τ , which corresponds to a straightforward application of the simplex learning to FedAvg. After the subregion assignment, FLOCO uses the degrees of freedom within the solution simplex to personalize clients models.We can further enhance the personalized FL performance of FLOCO by additionally fine-tuning a local model as in Ditto [16] . In this extension, called FLOCO + , each client personalizes the globalendpoints { θ 0 m = θ m } M m=1 by local gradient descent to minimize the Ditto objective, i.e., { θ k m } = argmin {θm} F k ({θ m }, { θ 0 m }) ≡ E α∼U Rz k [F k (w α ({θ m }))] + λ 2 M +1 m=1 ∥θ m -θ 0 m ∥ 2 2 .With the trained endpoints{ θ m = θ T m } M +1 m=1 , we simply use w α0 ({ θ m } M +1 m=1) as the global model, where α 0 = 1 M +1 1 M +1 with 1 D denoting the D-dimensional all one vector. For local models, we use{w α k ({ θ m } M +1 m=1 )} K k=1 where α k = α k .For FLOCO + , we fine-tune the corresponding subspace regions R z k for E local epochs.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: Our paper does not include any theoretic assumption or proof.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section, we experimentally show the advantages of FLOCO and FLOCO + over the baselines.Datasets and models. To evaluate our method, we perform image classification on the CIFAR-10 [33] and FEMNIST [34] datasets. For CIFAR-10, we train a CNN (CifarCNN) from scratch, following [35] , and fine-tune a ResNet-18 [36] pre-trained on ImageNet [37] , as in [38] . For FEMNIST, we train a CNN (FemnistCNN) from scratch, as in [1] , and fine-tune a SqueezeNet [39] pre-trained on ImageNet, following [38] . We provide a table with the training hyperparameters that we use for each dataset/model setting in Appendix B.Data heterogeneity for non-FL benchmarks. The FEMNIST dataset is an FL benchmark based on real data, where client heterogeneity is inherently embedded in the dataset. For CIFAR-10, we simulate statistical heterogeneity by two partitioning procedures. The first procedure by [40] partitions clients in equally sized groups and assigns each group a set of primary classes. Every client gets q % of its data from its group's primary classes and (100 -q) % from the remaining classes. We apply this method with q = 80 for five groups and refer to this split as 5-Fold. For example, in CIFAR-10 5-Fold, 20 % of the clients get assigned 80 % samples from classes 1-2 and 20 % from classes 3-10. The second procedure, inspired by [41] and [42] , draws the multinomial parameters of the client distributions p k (y) = Multi(y; ϕ k ) from Dirichlet, i.e., ϕ k ∼ Dir L (β), where β is the concentration parameter controlling the sparsity and heterogeneity-β → ∞ concentrates the mass to the uniform distribution (and thus homogeneous), while small 0 < β < 1 generates sparse and heterogeneous non-IID client distributions.Baseline methods. Besides FedAvg [1] and FedProx [21] for global FL, we chose FedRoD [14] , APFL [15] , Ditto [16] , and FedPer [17] as state-of-the-art personalized FL baselines.FLOCO Hyperparameters. For CifarCNN on the simulated non-IID splits Dir(0.3)/Five-Fold, we set τ = 250, M = 20/10, ρ = 0.1. For FemnistCNN on FEMNIST we set τ = 250, M = 10, ρ = 0.5. For pre-trained ResNet-18 on the simulated non-IID splits Dir(0.3)/Five-Fold we set τ = 50, M = 20/10, ρ = 0.1 and for the pre-trained SqueezeNet on FEMNIST we set τ = 250, M = 3, ρ = 0.5. We found those settings work well in our preliminary experiments, and conducted ablation study with other parameter settings in Appendix D. For the baselines, we follow the recommended parameter settings by the authors, which are detailed in Appendix B. Evaluation criteria. For the performance evaluation, we adopt two metrics, the test accuracy measured after the last communication round (ACC) and the time-to-best-accuracy (TTA), each for evaluating the global and local FL performance. ACC is the last test accuracy over T communication rounds, i.e, ACC(T ) = 1 Ntest Ntest i=1 1(y i = argmax g(x i ; w T )), where 1(•) is the indicator function that equals to 1 if the event is true and 0 otherwise. TTA evaluates the number of communication rounds needed to achieve the best baseline (FedAvg and Ditto in this paper) test accuracy, i.e., ACC FedAvg (T ). We report TTA improvement, i.e. the TTA of the baseline, e.g. FedAvg, divided by the TTA of the benchmarked method, e.g. FLOCO. Moreover, we report the expected-calibration-error (ECE) [43] , a common measure that evaluates the quality of uncertainty estimation of a trained model, for the last communication round.Table 1 and 2 summarize the main experimental results, where FLOCO and FLOCO + consistently outperform the baselines across the different experiments in terms of global (red) and local (blue) test accuracy, as well as test ECE. The global and local test metrics are measured after the last communication round and averaged over 5 different seed runs. The best performances are highlighted in bold, while the underlined entries indicate the settings that did not converge properly. Note that the global test performances of FEDAVG and DITTO, as well as FLOCO and FLOCO + , are the same since they use the same global model. Below we report on detailed observations. Global and local FL test accuracy. We first evaluate the global and local test performance on CIFAR-10 with the non-IID data splits generated by the 5-Fold and Dir(β) procedures, as well as the natural non-IID data splits in the FEMNIST dataset. Table 1 shows the test accuracies on CIFAR-10 with CifarCNN trained from random initialization (left) and ResNet-18 fine-tuned from the ImageNet pre-trained model (center), respectively. It also shows the test accuracies on FEMNIST with FemnistCNN trained from random initialization (left) and SqueezeNet fine-tuned from the ImageNet pre-trained model (right). We clearly see that FLOCO and FLOCO + outperform all baselines in terms of average local (blue) test accuracy by up to 6%, as well as global (red) by up to 5%.Calibration. We evaluate and benchmark the quality of uncertainty estimation of all methods. For this purpose we evaluate the global as well as average local ECE on each model-dataset combination for each baseline on the test dataset and show the results in Table 2 . As shown, FLOCO and FLOCO + achieve better Expected Calibration Error (ECE) across all settings, with two exceptions: training a pre-trained ResNet-18 on the CIFAR-10 Dir(0.3) split and a pre-trained SqueezeNetV1 on FEMNIST. In the first case, the average local ECE for FLOCO and FLOCO + is slightly worse than that of FedPer, suggesting mild overconfident for some clients. In the second case, the next best method (APFL) yields a significantly lower global test accuracy than our method, making a fair comparison of their ECE difficult.Worst client performance. We evaluate the average local and global test accuracies of the worst 5% of clients, a standard approach for assessing potential biases of the FL method toward specific clients or client groups [44] . The worst 5% client performance on all CIFAR-10/model combinations is evaluated over 5 trial runs, with results shown in the table on the right. We observe that FLOCO achieves the highest performance among worstperforming clients across all settings, with a 17% improvement over FedAvg, and up to 1.5% over the next best baseline. 1 , we plot the TTA improvement for FLOCO. In particular, we show the TTA improvement of FLOCO over FedAvg and FedProx, and the TTA improvement of FLOCO + over Ditto, FedPer and FedRod, as all these methods include local fine-tuning. We report all TTAs in Table 4 . The underlined entries indicate the cases where the test accuracies of our methods exceed the baseline method's maximum accuracy already at the initial evaluation round, while the entries labeled 'x1.0' represent the instances where our methods take the same evaluation rounds to achieve the baseline method's maximum accuracy, i.e., comparable in terms of TTA. In addition to test accuracy, we also observe an improvement in Time-to-Accuracy (TTA) for our method across all settings. In this section, we provide further analyses and discussion on FLOCO.Solution structure in simplex. First, we confirm that FLOCO uses the degrees of freedom within the solution simplex for personalization. To this end, we draw approximately 500 uniformly distributed points in the solution simplex, and evaluate the global and the local test accuracy of the corresponding models. For FLOCO, we can clearly observe a jump in average local test accuracy at τ = 250, which is a result of our subregion assignment. Right shows the total variance of the gradients for the last fully-connected layer.Gradient variance reduction and stability of training. Figure 2 shows the test accuracy curves during training for global (left) and average local (center) test accuracies of different methods with the standard deviation over 5 trials as shadows. We observe that FLOCO and FLOCO + not only converge faster than the global and pFL baselines respectively, but also show small standard deviation across trials. The latter implies that our systematic regularization through the solution simplex stabilizes the training dynamics significantly. Figure 2 (right) shows the total gradient variance-the sum of the variances of the updates ∆w t k = w t k -w t-1 0 for FedAvg and FedProx (which almost overlap with each other), and ∆θ t m,k = θ t m,k -θ t-1 m,0 for FLOCO, respectively. More specifically, we compute the variance over the last fully-connected layer, given by σ 2 total (t) = k∈S t ∥∆w t k -1EQUATION)for FedAvg and FedProx, and byEQUATION)We have not plotted the gradient variances of FLOCO + and the other pFL methods, since those are the same as for FLOCO and FEDAVG, respectively. As discussed in [45, 46] , a small total variance indicates effective collaborations with consistent gradient signals between the clients, leading to better performance. From the figure, we see that the total gradient variance of FLOCO is much lower and more stable, in terms of standard deviation, than the baseline methods, which, together with its good performance observed in Table 1 , is consistent with their discussion. The variance reduction with FLOCO implies that the degrees of freedom of the solution simplex can absorb the heterogeneity of clients to some extent, making the gradient signals more homogeneous. Moreover, [47] argued that the last classification layer has the biggest impact on performance, implying that reducing the total variance of the classification layer, as FLOCO does with simplex learning, is most effective. As we show in the Appendix C, applying simplex learning to only the last layer, instead of learning a simplex in the whole parameter space, achieves faster personalized and global convergence.Computational complexity. If the batch size is one, simplex training adds O(π • M ) computational complexity for each layer, where π is the parameter complexity of the layer, e.g., π = d • L for a fully connected layer with d input and L output neurons, and M is the simplex dimension [30] .For FLOCO, this additional complexity only applies to the classification layer. For inference, no additional complexity arises, compared to FedAvg, because inference is performed by the single model corresponding to the cluster center. Since the most modern architectures, e.g., ResNet-18 and Vision Transformer (ViT) [48] , have parameter complexity of O(G FE ) ≫ O(G C ), where G FE and G C are the complexities of the feature extractor and the classification layer, respectively, the additional training complexity, applied only to the classification layer, of FLOCO is ignorable, i.e., O(G FE ) ≫ O(G C • M ). The same applies to the communication costs: since the simplex learning is applied only to the classification layer, the increase of communication costs are ignorable compared to the communication costs for the feature extractor.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Our paper gives detailed information on how to reproduce our results, including all hyperparameters, models and dataset splits needed as well as a detailed description for our algorithm. Moreover, we upload our code together with the submission which includes a README that documents how experiments can be reproduced. 
related work: 
 There are few existing works that apply simplex learning to federated learning. [35] proposed SuPerFed, which enforces a low loss simplex between independently initialized global and client models, yielding good personalized FL performance. This approach builds on [25] , which finds optimal interpolation coefficients between a global and local model to improve personalized FL. However, their simplex is restricted to be 1D, i.e., a line segment, and the global model performance is comparable to the plain FedAvg. Moreover, they train a solution simplex over all layers between global and local models, which is computationally expensive and limits its applicability to training from scratch. This should be avoided if pre-trained models are available [38, 49] . Our method generalizes to training low-loss simplices of higher dimensions in a FL setting, tackles both the global and personalized FL objectives, is applicable to pre-trained models, and shows significant performance gains by employing our proposed subregion assignment procedure. In Table 7 of Appendix E we benchmark FLOCO against the SuPerFed baseline on the CIFAR-10, 5-Fold, as well as Dir(0.5) splits using both a CifarCNN trained from scratch as well as a pre-trained ResNet18 on both global as well as local test performance, where we observe that FLOCO outperforms SuPerFed both in terms of global as well as local accuracy in all settings.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide our code in the submission which includes a README with detailed description on how to run our method in order to reproduce the shown results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards 
limitations: 
 In this work, we only evaluate our method on cross-silo FL settings with up to 100 clients. Unlike cross-device FL, which typically involves a much larger set of stateless clients (i.e., clients with limited data that hinders reliable modeling), our approach assumes stateful clients, each with sufficient data to enable effective grouping of similar clients. While our current analysis focuses on cross-silo FL, extending our method to the cross-device setting is an important direction for future research. Additionally, a thorough theoretical analysis of our approach remains a future research objective. 
conclusion: 
 FL on highly non-IID client data distributions remains a challenging problem and a very actively researched topic. Recent works tackle non-IID FL settings either through global or personalized FL. While the former aims to find a single optimal set of parameters that fit a global objective, the latter tries to optimize multiple local models each of which fits the local distribution well. These two different objectives may pose a trade-off, that is, personalized FL might adapt models to strongly to local distributions which might harm the global performance, while global FL solutions might fit none of the local distributions if the local distributions are diverse. In this paper, we addressed this issue by leveraging the mode-connectivity of neural networks. Specifically, we propose FLOCO, where each client trains an assigned subregion within the solution simplex, which allows for personalization, and at the same, contributes to learning a well-performing global model. FLOCO achieves state-of-the-art performance in both global and personalized FL, with minimal computational and communication overhead during training and no overhead during inference.Promising future research directions include better understanding the decision-making process of solution simplex training through global and local explainable AI methods [50] [51] [52] . Furthermore, we want to apply our approach to continual learning problems and FL scenarios with highly varying client availability [53, 54] .ResNet-18 with learning rate of batch size 32, using SGD with a learning rate of 0.01, momentum of 0.9, and weight decay of 10 -4 . For FEMNIST we train a pre-trained SqueezeNet with batch size 32 using SGD with a learning rate of 0.005, momentum of 0, weight decay of 10 -4 , and a FemnistCNN with batch size 32, learning rate 0.1, momentum of 0, weight decay of 0. For FedProx we set the proximity hyperparameter to µ = 0.01 for all settings. For DITTO, FEDROD and FEDPER we set the local epochs to the same value as epochs for the global model, i.e. E DITTO = E. All training hyperparameters for CIFAR-10 and FEMNIST on a FemnistCNN were taken from [35] , CIFAR-10 on a pre-trained ResNet-18 from [49] and FEMNIST on pre-trained SqueezeNet from [38] .  
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Figure 5 , we compare the global (left) and average client (right) test accuracy of FLOCO and FLOCO-All, where the latter applies simplex learning to all NN parameters. As expected, FLOCO-All converges to the same global and average local test accuracy, but needs more communication rounds to do so, since it needs to train more parameters. We investigate how stable the performance of FLOCO is for different hyperparameter settings. Specifically, we tested FLOCO with the combination of τ = 50, 100, 200 (subregion assignment time step) and ρ = 0.1, 0.2, 0.4 (radius of subregions), and show the average local client and global test accuracy for CifarCNN on CIFAR-10 5-Fold in Figure 6 . We observe that the average local client test accuracy (left) increases for earlier subregion assignment starting points τ and lower client subregion radiuses ρ, with the best reached test accuracy being approximately 4% better than the worst, i.e., 82.79% against 78.18%. The intuition for this is that earlier client specialization in less overlapping regions allows for better personalization. On the other hand, as can be observed in the right heatmap of Figure 6 the global test performance is less sensitive to the choice of these hyperparameters, i.e., 70.66% against 69.30%. This is because, even after subregion assignment, the entire solution simplex remains to be trained, making the midpoint (global model) of the simplex less sensitive to the specialization process for client distributions.= 0.02 = 0.1 = 0. We benchmark FLOCO against the SuPerFed baseline on the CIFAR-10, 5-Fold, as well as Dir(0.5) splits using both a CifarCNN trained from scratch as well as a pre-trained ResNet18, on both global as well as local test performance. As shown in Table 7 , FLOCO outperforms SuPerFed in all settings. Note, that for this benchmark we have implemented FLOCO as well as SUPERFED in the FL framework Flower [19] .Justification: We show that, in some cases, our method does not exceed the performance of a baseline method, however, we show that it saves up computational cost which is very relevant in the field. We discuss this point in more detail and give an alternative solution.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Note, that we do not rely on any regularizer that forces the diversity of the endpoints, as in[30]. In FLOCO, the diversity of local client distributions prevents the simplex endpoints from collapsing to a single point. 


Decision: **Summary:**

FLOCO proposes using linear mode connectivity to identify a solution simplex in neural network weight space for federated learning. Clients are assigned subregions in the simplex based on gradient signals to allow personalization while jointly learning a global solution. Claims to accelerate global training and improve local accuracy compared to baselines.

**Strengths:**

- Novel application of linear mode connectivity/simplex learning to federated learning
- Well-written paper with comprehensive experiments
- Addresses both global and personalized FL in a unified framework
- Achieves good performance on both global and local accuracy metrics

**Weaknesses:**

- Limited theoretical analysis
- Some baseline comparisons missing initially (addressed in rebuttal)
- Scalability to very large numbers of clients unclear
- Applicability may be limited to cross-silo rather than cross-device FL settings

**Key Discussion Points:**

- *Baseline comparisons:* Authors added several new baselines in rebuttal, showing FLOCO outperforms them
- *Applicability:* Authors acknowledged FLOCO is more suited for cross-silo FL with stateful clients, not cross-device scenarios
- *Scalability:* Questions raised about performance with thousands of clients; authors agreed this is a limitation for low-data clients
- *Ablation studies:* Additional experiments requested and provided to isolate effects of different components
- *Theoretical analysis:* Lack of convergence analysis noted; authors agreed this is future work
- *Implementation details:* Clarifications provided on simplex learning process

Overall, reviewers generally found the core idea interesting and results promising, but had some concerns about scope/applicability and analysis depth. Scores ranged from borderline reject to weak accept, with most leaning positive after author responses. The consensus seems to be that this is a solid contribution to federated learning research, particularly for cross-silo scenarios, though some limitations remain.

Reviews: review: paper_summary: paper_summary: The authors propose federated learning over connected modes (FLOCO), where clients are assigned local subregions in this simplex based on their gradient signals, and together learn the shared global solution simplex.
strength_weakness: strength_weakness: The proposed methodology lacks innovation.

The experimental baseline is too old (2021) and there are many recent and similar clustered federated algorithms that lack comparison.
questions: questions: 1 Can the authors clarify the difference between the proposed method and other clustering federated learning algorithms?

2 Can you specifically describe the implementation details of the algorithm? Or improve the description. For example, there is a lack of more detailed description for the generation of R.

3 In the algorithm section, can the specific algorithm be explained more clearly?
limitations: limitations: The authors discuss the reasons for the poor experimental performance of the proposed methods
paper_summary: paper_summary: A novel method is proposed to derive the mode connectivity over the simplex defined by the central server for an improved global model as well as local personalization performances in the federated settings.
strength_weakness: strength_weakness: - Line 25: is leverages -> leverages
- Line 26, 33, 66, 129: weight parameter -> parameter (since parameters typically represents both weights and biases in NNs)
- Line 60: over the clients -> for the $k$-th client
- Line 96: (not sure) in the RHS, it may be $\boldsymbol{w}_\boldsymbol{\alpha}$
- Line 127: please consider adding equation numbers in each line of the pseudocode of Algorithm 1. 
- Line 148: missing citation
- Lines 189-190: the baseline methods are not convincing. Since the proposed method explicitly induces the mode connectivity (i.e., line 25 per se) for the improved FL performance, it should have been compared to methods with similar motivations, e.g. [1], which was cited in the Related Works section (line 286) but not directly compared. Plus, authors should have also considered adding more personalization methods (e.g., [2], which shares the same strategy, i.e., only exchanging parameters of classification layer, for communication-efficient personalization in FL) as the authors aim to prove the effectiveness of `FLOCO` for improved personalization performances. 
- Line 212: please also report standard deviations on 5 different runs. 
- Lines 461-466: the communication rounds (100 and 500), the number of clients (100), and the number of per-round clients in CIFAR-10 (which is 30 out of 100) are somewhat deviated from the practical FL settings, which raises doubts about the scalability of the proposed method. It is more acceptable if the proposed method is specifically designed for the cross-silo FL settings, which usually assume a small number of stateful clients and (optionally) full participation across rounds. 


[1] Connecting low-loss subspace for personalized federated learning (2022)  
[2] Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020)
questions: questions: - Line 255: in Figure 2, the convergence speed of the global test accuracy & the average local test accuracy of the proposed method is slower than the baseline methods (when seeing round ~50), which is somewhat counterintuitive provided that the total gradient variance remains the smallest across all FL training rounds (which authors stated that it leads to the better performance, i.e., line 265-266). Could the authors please provide an explanation on this?
- Line 255: same in Figure 2 (and similarly in Figure 3), the average local test accuracy is suddenly surged, starting from the 200th round. This is quite intriguing, and I also expect an explanation on this result.
limitations: limitations: The authors do not evaluate the proposed method out of vision classification task.
paper_summary: paper_summary: In this study, the authors tackle challenges in federated learning by introducing FLOCO, which uses linear mode connectivity to identify a solution simplex in neural network weight space. This approach allows for personalized client model training within the simplex, while also enabling efficient updates to both global and local models.
strength_weakness: strength_weakness: To strengthen their empirical findings, the authors could benefit from leveraging the bilevel optimization literature for analyzing the convergence of their algorithm.
questions: questions: $\textbf{Q1}.$  Choosing an appropriate experimental setup is often overlooked in the pFL setting. Note that the following two scenarios are inappropriate for pFL experiments: $\textbf{(1)}$ Clients have enough data to train their local models independently, eliminating the need to participate in an FL system. $\textbf{(2)}$ The data distribution of the clients is identical or nearly identical. This typically results in high accuracy for FedAvg, making it challenging to justify using pFL methods. It would be informative to evaluate your method in experimental setups where local training and FedAvg fail to provide good results and compare your method's performance with SOTA  pFL methods, especially methods with decoupled parameter space such as FedPer [1] and FedRep [2] and similarity-based methods such as pFedSim [3].

$\textbf{Q2}.$ As previously mentioned, evaluating the performance of local models on their respective local datasets is crucial in personalized federated learning (pFL). Including the performance metrics of these models (average performance when trained solely on local training data) would provide valuable insights into the experimental results.

[1] Arivazhagan, Manoj Ghuhan, et al. "Federated learning with personalization layers." arXiv preprint arXiv:1912.00818 (2019).

[2] Collins, Liam, et al. "Exploiting shared representations for personalized federated learning." International conference on machine learning. PMLR, 2021.

[3] Chen, Yizhu, et al. "PFedSim: An Efficient Federated Control Method for Clustered Training." IEEE Journal of Radio Frequency Identification 6 (2022): 779-782.
limitations: limitations: The discussion of the limitations of the proposed method is framed within the context of future directions. It is suggested to address these limitations separately, focusing on theoretical analysis and validation using real-world federated learning datasets suited for personalized FL (pFL).
paper_summary: paper_summary: This paper proposes a method for FL to train a simplex of linearly connected solutions with uniformly low loss. Clients are expressed in the simplex by projecting the gradient onto the simplex and similar clients are close to each other. In each communication round each client samples points in the neighborhood of their projected point and jointly train the solution simplex.

Each client is expressed as a point in the simplex based on the gradient update signals. The subregion assignments preserve the similarity of the clients by applying the Euclidean projection onto the positive simples with Riesz s-Energy regularization - this spreads the projections while maintaining the similarity between the clients. As the gradient update signals are informative for the subregion assignment only after the model has been trained for some time, the first stage of the algorithm is FedAvg with simplex learning.
strength_weakness: strength_weakness: Some details of the algorithm should be better explained. Some baselines seem to be missing.
questions: questions: - Could you clarify how simplex learning works in practice? Do you compute the loss and the gradients on each of the endpoints or do you sample one point in the simplex and backpropagate to update the endpoints? On which weights do you compute the loss? I think this is unclear from the paper and it would be beneficial to add a section “Simplex Learning” for the reader.
- From my understanding, the idea of using a shared simplex and then dividing it into subregions is to have both global and local classifiers and to incentivize knowledge transfer and robustness. I would also see two other baselines/ablations that the authors might want to try to corroborate that this is actually beneficial. First, fedavg simplex learning (no subregions, check global accuracy) and then local fine-tuning for personalization (compute local accuracy) personalization can be done with local simplex learning or standard learning. I guess this could be a way to study the impact of simplex learning by itself and the importance of subregions.
- As the authors mentioned in the related work, there are other personalized FL methods that apply simplex learning. Why did the authors not report their results as baseline? E.g. references 29 and 19.
- I’m not sure I fully buy the claims and observations in the paragraph on variance reduction (lines 255-272). The main issue with heterogeneity is convergence speed, so to claim some effect on this the authors should show faster convergence on the global accuracy. There is evidence that the effect of the subregions impacts local accuracy, but this is expected as there is better specialization - as it would be with a local fine-tuning stage.
- I suggest the authors also have a look at other more recent methods for global learning such as scaffold, feddyn, mime that could help with heterogeneity. They could also be used in conjunction with simplex learning.
- I found it odd that the authors decided to show the plots in Figure 2 using the simpleCNN and FEMNIST while from table 1 the cifar10 experiments show more edge of FLOCO over the baselines.
- Basically, the method only applies to the last FC layer - did the authors try it on the full model?
limitations: limitations: Limitations are not clearly expanded and should be discussed in a section of the paper.

Comments: title: DITTO
Reply to authors
Practicality
Response?
Start discussions with authors, please
Reply to authors
Extension
comment: Thank you - this helps.
FedAve + DITTO personalization corresponds to the baseline "DITTO" in the original submission, of which the results are reported in Fig. 2 and Tables 1-3.
Thank you - I will take some time during the reviewers discussion to take into account these results. I think I don’t see the baseline with standard fedavg and personalization, am I correct ?
Here we report on the average local test accuracy of Baseline 1, Baseline 2, and FLOCO+. Additionally, we tested a new version of our method, FLOCO++, inspired by our observation that Baseline 1 was better than Baseline 2.


- ***FLOCO++:*** We run FLOCO (with client projection after 100 communication rounds), and apply the DITTO personalization to the projected point of each client.


As we mentioned in the previous comments, we used a (M=2)-simplex for a SimpleCNN on CIFAR10 with a Dirichlet(0.5) split for computational reasons (our cluster server is down until the weekend).  Below we report on the average local test accuracy at 100, 200, 300, 400, and 500 communication rounds.


- ***Baseline 1*** (Global simplex learning + plain DITTO personalization to the midpoint):
68.92, 74.09, 76.05, 76.92, 77.02


- ***Baseline 2*** (Global simplex learning + DITTO personalization with local simplex):
69.43, 74.08, 76.10, 76.96, 76.86


- ***FLOCO+:***
69.60, 74.20, 76.10, 77.00, ***77.17***


- ***FLOCO++:***
70.01, 74.32, 76.11, 77.08, ***77.41***


As expected, the personalization boosts the local test accuracy in general, and the gain by FLOCO is reduced, compared to what we showed in the rebuttal pdf as Fig. 4. However, we still observe some gains by FLOCO’s projection strategy. We expect that the comparison with M=6 would show larger gains by our methods, since a larger space inside the simplex should better capture the characteristics of the clients.  We will conduct the experiment with M=6 next week when our cluster servers will work again, and use the results in the paper.  We will also investigate why the local simplex DITTO personalization (Baseline 2 and FLOCO+) is worse than the plain DITTO personalization (Baseline 1 and FLOCO++). Thank you for your patience in waiting for our experimental results until the last minutes, and thank you again for your valuable comments to improve the paper. We will add FLOCO++ as another version of our proposal in the paper.
Thank you for your answer. I’ll wait for the experiments. As I mentioned in my review I think these are important to better understand how the method works: 

> From my understanding, the idea of using a shared simplex and then dividing it into subregions is to have both global and local classifiers and to incentivize knowledge transfer and robustness
Dear Reviewer,

Thank you very much for your response. Since the reviewer said that our submission is a borderline paper and the reviewer would keep the original score "3: Reject", we wonder if the reviewer perhaps misunderstands the scoring system of NeurIPS. The NeurIPS review system adopts the scoring system NOT 1-5 but 1-10, and the score "3: Reject" that the reviewer gave to our paper is not for the borderline papers, as the reviewer assessed our paper in the previous response, but for the papers that should be rejected. Therefore, we would kindly ask the reviewer to adjust the score, keeping in mind the scoring system that the NeurIPS committee has defined, i.e., 2: Strong reject, 3: Reject, 4: Borderline Reject, 5: Borderline Accept, 6: Weak Accept.

Additionally, if the reviewer still leans toward a reject, giving, say score 4: Borderline Reject, we would kindly ask the reviewer to provide the reasons for rejection by explaining which of our responses is not sufficient, and more importantly, why.

Kind regards, 
Authors
I sincerely appreciate the authors' positive answers.  
Someone might think that narrowing the scope of the work limits the scalability of one's proposed method, but I humbly believe that clarifying what can and can't be done in one's work adds more value to the research community as well as clarifies its practical implications.  
Thus, please reflect what the authors have agreed upon during discussion periods in their final manuscript.  
In conclusion, I decide to raise my score, and here are reasons.

### Why not a lower score
* The authors made enough efforts in rebuttals to complement lacking and outdated baselines in their initial manuscript, and the results are convincing.
* After the discussion, the authors decide to adjust the main scope of their work in a more acceptable direction.
* The observation of the mitigating gradient variance in the proposed method (i.e., Fig. 2) is also intriguing and noteworthy.

### Why not a higher score
* The scope (and possibly the title) of the manuscript should be refined according to the authors' acceptance of my suggestions (i.e., specifying the scope of the algorithm in a cross-silo FL setting), which will require some changes (e.g., re-writing of overall contents) to the status quo.  
* While I recognize and accept that the mode connectivity --- a main motivation and an ingredient of the proposed method --- is fairly an empirical observation, a theoretical or in-depth empirical analysis (e.g., generalization guarantee, or convergence analysis) that incorporates the mode connectivity for federated settings is missing. The authors could complement this perspective by building on the recent analysis. (e.g., [1,2])
* (minor) Overall, the notations are difficult to parse and look similar to each other at first glance.

[1] Proving Linear Mode Connectivity of Neural Networks via Optimal Transport (AISTATS'24)  
[2] Linear Connectivity Reveals Generalization Strategies (ICLR'23)
Thank you for the rebuttal. After reading the rebuttal and reviews of other reviewers, I still maintain that this is a borderline paper. I will maintain my original rate.
Dear Reviewer a16f,

Thank you very much for your additional comments.  Below we give temporal answers, and we will report on additional experiments with the suggested baselines by the discussion deadline.

Best,
Authors


***Reviewer’s question:*** In Q2 I've suggested more than one option for personalization. Could you clarify how you implemented the experiments for this ablation? I don't understand if the simplex baseline implies some personalization (local fine-tuning).

***Author’s answer:*** We realized that we misunderstood your suggestions when submitting our rebuttal.  The curves labeled as FedAvg-Simplex Learning in Figs. 3 and 4 in the rebuttal pdf are the global (Fig.3) and the average local (Fig.4) test accuracy of simplex learning without client projection and WITHOUT personalization.  We apologize for our confusion. We are now evaluating the following methods and will report on the average local test accuracy by the discussion deadline:

- Baseline 1: We run FedAvg with simplex learning, and then, apply the plain DITTO personalization to the midpoint of the simplex solution. Namely, no local simplex is used.
- Baseline 2: We run FedAvg with simplex learning, and then, apply the DITTO personalization with local simplex learning.
- FLOCO+: We run FLOCO (with client projection after 100 communication rounds), and apply the DITTO personalization with local simplex learning.

Unfortunately, the cluster server in my group is under maintenance after the Neurips rebuttal phase ended, and we are running the additional experiments on local computers.  To make the experiments feasible, we have to down scale the experiment.  Namely, our simplex learning is with a (M=2)-simplex (instead of M=6 in the submitted paper) for a SimpleCNN on CIFAR10 with a Dirichlet(0.5) split.  To make a fair comparison with our FLOCO+, we will also report on FLOCO+ results with M=2-simplex.  We estimate that all experiments will finish by tomorrow (Tuesday) night.


***Reviewer’s question:*** Regarding Q4, I understand what you mean by variance reduction now, but in the context of FL and optimization, this has a specific meaning (see SCAFFOLD). By checking again table 1 I see the improvement over simple baselines such as fedavg also in global accuracy. However, since the baselines for global models are not the most recent ones (besides the newly tested feddyn, which is generally broken) I suggest the authors tone down their claims in the contributions regarding "FLOCO outperforms state-of-the-art approaches in both global and local". It is however interesting the effect of their method on global accuracy hence reducing the effect of heterogeneity.

***Author’s answer:*** We agree that our original claim of outperforming state-of-the-art global FL approaches is not supported by our experiments.  Following the reviewer’s suggestion, we will tone down our claim.


***Reviewer’s question:*** One last comment is about scalability. Could you comment on the possibility of making your method work in more realistic settings with thousands of clients (e.g. inaturalist, gldv2) ? What are the challenges you foresee? this could be expanded in the limitations section.

***Author’s answer:*** We assume that by scalability the reviewer means the situation where many clients do not observe sufficient numbers of  data samples.  If we would have thousands of clients, all of which observe sufficiently many data samples, we do not foresee critical problems because the projection matrix can be computed with subsamples, and the other computations are linear operations.  However, if we assume the generic FL setting, we cannot reasonably assign subregions to the low data clients, because the server does not get sufficient information about those clients.  Following the suggestions by Reviewer a16f, as well as Reviewer 44Bh, we will revise the paper so that our target is not the generic FL setting, but the cross-silo FL setting, where each client observes sufficiently many data samples. We appreciate the reviewer’s constructive suggestions.
Dear Reviewer bjGi,

Thank you again for your constructive comments.  We will do our best to improve the paper, according to the reviewer’s comments.

Best,
Authors
Dear Reviewer 44Bh,

Thank you for your additional comments.  We agree with the reviewer that we only tested our method on the sampled 100 clients in FEMNIST.  As the reviewer pointed out, our method, in particular in the subregion assignment, requires the statefulness of the clients. In the generic FL setting where many clients with few data samples exist, the projection and the subregion assignment in FLOCO are not reliable anymore.  Therefore, we expect that the low data clients are assigned random subregions, which will deteriorate the performance.  Following the suggestion by Reviewer 44Bh, as well as by Reviewer a16f, we revise the paper so that our target is not the generic FL setting, but the cross-silo FL setting, where each client observes sufficiently many data samples.  We appreciate the reviewer’s constructive suggestions.

Best,
Authors
Dear Reviewer bjGi,

Since the discussion deadline is approaching, we would kindly ask Reviewer bjGi to respond to our rebuttal comments.  We see that our rebuttal comments addressed most of the criticisms by the reviewer, and would like to know the reasons why the reviewer still keeps the clear rejection score 3 after our rebuttal, which is a clear outlier among the other reviewers’ scores.  We look forward to your response.

Best,
Authors
Thank you for your rebuttal. I would like to maintain my current score.
I second Rev 44Bh comments. As mentioned in my response, I also think the authors should comment on the scalability of their algorithm and the applicability on general cross-device scenarios. Looking forward to the authors response.
Dear Reviewers,

Thank you for your valuable contributions to the review process. As we enter the discussion phase (August 7-13), I kindly request your active participation in addressing the authors' rebuttals and engaging in constructive dialogue.

Please:

- Carefully read the authors' global rebuttal and individual responses to each review.

- Respond to specific questions or points raised by the authors, especially those requiring further clarification from you.

- Engage in open discussions about the paper's strengths, weaknesses, and potential improvements.

- Be prompt in your responses to facilitate a meaningful exchange within the given timeframe.

- Maintain objectivity and professionalism in all communications.

If you have any concerns or need guidance during this process, please don't hesitate to reach out to me.

Your continued engagement is crucial for ensuring a fair and thorough evaluation process. 
Thank you for your dedication to maintaining the high standards of NeurIPS.

Best regards,

Area Chair
Thanks for the authors' time and effort for my suggestions and questions.  
Especially, I really appreciate additional experiments that authors have committed to improve their initial manuscript.
 
I have only one remaining concern that needs to be clarified further.
> The question about the practicality of the current experimental setting & the authors' answer about the FEMNIST baseline being used.

While it is acceptable that FEMNIST dataset[1] if $\texttt{Floco}$ is designed for real-world _cross-device_ federated setting, it seems that the authors did not use the full dataset as stated in Appendix B, line 464.  
In other words, the total number of clients in FEMNIST dataset is originally 3,597, but only 100 clients are sampled and used in this work.  
That is why I recommended authors to tone down and specify the scope of this work into __cross-silo__ FL setting, rather than sticking to an algorithm for generic FL setting.   

This can be further justified due to the design choice of the proposed method:
* The $\mathcal{R}_k$ should be __kept and tracked__ by the server for each client $k\in[K]$ till the end of the federation round.
* This means that the algorithm requires __stateful__ clients, which does not hold in the practical _cross-device_ setting. (please see Table 1 of [3])
* Moreover, this statefulness inevitably forces the algorithm to have a moderate number of clients, $K$, which is consistent with the current choice of experimental settings.

If I have misunerstood and if the proposed method is even scalable to the empirical settings with massive number of clients (e.g., original FEMNIST setting with $K=3,597$, StackOverflow dataset[2] with $K=342,477$, and iNaturalist dataset[4] with $K=9,275$), please enlighten me.  

[1] (2019) LEAF: A Benchmark for Federated Settings  
[2] (2019) https://www.tensorflow.org/federated/api_docs/python/tff/simulation/datasets/stackoverflow/load_data  
[3] (2019) Advances and Open Problems in Federated Learning  
[4] (2020) Federated Visual Classification with Real-World Data Distribution
Thank you for your rebuttal. I appreciate your effort and will take into account all your responses in my final score which I want to emphasize is positive.  
- In Q2 I've suggested more than one option for personalization. Could you clarify how you implemented the experiments for this ablation? I don't understand if the simplex baseline is implying some personalization (local fine-tuning) or not.
- Regarding Q4, I understand what you mean by variance reduction now, but in the context of FL and optimization, this has a specific meaning (see SCAFFOLD).  By checking again table 1 I see the improvement over simple baselines such as fedavg also in global accuracy. However, since the baselines for global models are not the most recent ones (besides the newly tested feddyn, which is generally broken)  I suggest the authors tone down their claims in the contributions regarding "FLOCO outperforms state-of-the-art approaches in both global and local". It is however interesting the effect of their method on global accuracy hence reducing the effect of heterogeneity.
- One last comment is about scalability. Could you comment on the possibility of making your method work in more realistic settings with thousands of clients (e.g. inaturalist, gldv2) ? What are the challenges you foresee? this could be expanded in the limitations section.
***Reviewer’s question 7:*** Basically, the method only applies to the last FC layer - did the authors try it on the full model?
***Author’s answer:*** We show the result with FLOCO-all in Figures 5 and 6 in the rebuttal pdf, where we observe that applying FLOCO to all layers significantly reduces training speed.