Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

Authors: minghui  chen
meirui  jiang
xin  zhang
qi  dou
zehua  wang
xiaoxiao  li
  kong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 09:33:40

Abstract: Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. 
Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. 
However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. 
To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.''
Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. 
This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL.
We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets.

Main paper: 
 introduction: 
 Federated learning (FL) [35] has emerged as a promising methodology for leveraging the power of private data without the need for centralized data governance. However, data heterogeneity in FL poses significant challenges to the design of efficient training for global convergence. With the emergence of the pre-training and fine-tuning paradigm in various applications [15, 19] , recent studies [37, 2] have attempted to address the problem of FL under data heterogeneity with pre-trained initialization. Although pre-trained federated learning can speed up convergence compared to random initialization, it still requires a significant number of communication rounds between the server and clients, often amounting to hundreds of rounds [37] . Existing pre-trained models [41, 47] often have an enormous parameter scale, and following the neural scaling law [24] , there is a continuous trend toward increasing model parameters. Deploying models with such a large parameter size in FL introduces significant communication overhead. This greatly hampers the flexibility and scalability of model updates. Reducing FL communication overhead can be approached by reducing the scale of model parameters involved in distributed training [59] or reducing communication rounds [35] . Comparing with reducing model parameters, reducing communication rounds typically leads to a more efficient reduction of network congestion [17] , decreased energy consumption on client devices [33] , and a lower risk of privacy breaches [61] . In this paper, we focus on reducing communication rounds in FL with pre-trained model as initialization.Typically, increasing the number of local training steps can effectively reduce communication rounds. However, there is an upper limit to the extent of local training step increments. This limitation arises due to the presence of data heterogeneity, where the consistency of optimization among different clients deteriorates with the increasing number of local steps [35] . This optimization inconsistency leads to a discrepancy between local and global models and decelerates the convergence rate of FL. The discrepancy is often called client drift [25] . Previously, some FL methods [25, 45] attempted to introduce proximal terms to regularize local training, with the aim of reducing local overfitting and minimizing the problem of client drift. While these methods can accelerate convergence, they restrict the progress of each local training steps towards the optimal solution, impeding the attainment of FL with more aggressive communication round reductions. While these client drift mitigation methods can reduce local overfitting to some extent, they cannot ensure strong performance of the global aggregated models, particularly in scenarios with limited communication rounds. This situation arises when individual local clients become trapped in isolated low-loss valleys. More specifically, as illustrated in Figure 1 , two models from clients 'A' and 'B', even if their optimal model distance is small, still result in a poorly performing aggregated global model. Moreover, the preceding FL methods aimed at minimizing communication rounds exclusively address scenarios involving random initialization, lacking a customized approach tailored to pre-trained models. Recent proposed centralized fine-tuning methods (e.g., model soups [52] and DiWA [43] -a greedy model selection version of model soups) based on model interpolation (averages of a large number of model weights) are effective approaches to seek large connected low-loss region, which are promising for applying in FL to reduce communication rounds. These methods can prevent individual clients from being trapped in isolated low-loss valleys by positioning the global model centrally within a larger low-loss region by overlapping the low-loss regions among clients, as shown in Fig. 1 (right). However, their training efficiency is exceedingly low, requiring complete retraining of numerous models, leading to significant computational overhead on clients and intolerable communication costs when applied in FL, due to two aspects: First, they involve a time-consuming model selection phase within the model pool, which consists of all candidate models available for weight interpolation. Secondly, model soups entail an extensive number of model training iterations, lacking prior guidance and relying on brute-force, random, and often redundant training. Many of the trained models end up unused.To enjoy the connected low-loss valley benefits of model soup-based methods [52, 43] without burdening local training, we propose an efficient and local model interpolation-based method, called Local Superior Soups (LSS). To address the first issue, we propose a sequential random model interpolation method during training. This eliminates the need for subsequent model selection steps and ensures that the models slated for interpolation reside within the same low-loss valley during training (Sec. 3.3.1) . For the second issue, we introduce two quantifiable indicators of candidate model quality, inspired by data augmentation quality quantification [32, 4] : diversity (Sec. 3.3.2) and affinity (Sec. 3.3.3) . Specifically, the diversity indicator quantifies the diversity among models in the model pool with their pairwise model distance, where larger distances denote higher diversity, signifying better model quality for interpolation. As illustrated in Figure 2 (left), a low-loss region, supported by models with low diversity, can be effectively covered with only a few candidate models positioned near its periphery. Thus, we propose incorporating the diversity metric as a regularization term during training to maximize the expansion of low-loss regions, thereby increasing the utilization of trained models. The affinity indicator measures the affinity of each candidate model in the model pool to the initial model. Smaller distances indicate greater affinity, indicating better model quality for interpolation. This affinity is also incorporated as a regularization term during training to prevent the expansion of low-loss regions from deviating too far from the shared initialization point, thus increasing the likelihood of overlapping connected regions (as depicted on the right side of Fig. 2 ). These two indicators facilitate the efficient inclusion of models into the model pool, preventing wasteful training of models that may ultimately go unused.HighIn experiments, we found that our proposed method greatly reduces communication rounds, and we achieved the performance of models fused after multiple rounds of communication in other FL methods with only a few rounds of communication.In summary, our contributions are as follows.(1) We reveal the importance of regularizing local client models in the connected low-loss valleys for reducing communication rounds when initializing FL with pre-trained models. (2) We introduce an innovative and efficient model soups-based method for FL, called Local Superior Soups (LSS) that eliminates the need for time-consuming model selection and redundant model training in the existing soups-based approaches, while expanding connected low-loss valleys of client models for faster convergence. (3) In experimental evaluations, LSS demonstrates a significant reduction in communication rounds, achieving superior performance with only a few rounds of communication, exceeding baseline FL methods significantly in four datasets and two types of distribution shifts.2 Related Work 
heterogeneous federated learning: 
 FL struggles with Non-IID data, leading to various proposed algorithms. FedProx [28] uses proximal term to regularize local training, preventing client divergence. Scaffold [25] adds variance reduction to combat "clients-drift." MOON [27] employs mode-level contrastive learning to stabilize local training.Personalized FL [46] targets high local performance on Non-IID data. FedBN [30] applies local batch normalization to mitigate feature shift before model averaging. Recent one-shot and few-round FL methods use parallel server-side techniques like prediction ensembles [14] , data generation [56, 18] , or meta-learning [39] to improve aggregated model performance.Fine-tuning leverages pre-trained models to enhance task performance [7] . FedFTG [57] proposes knowledge distillation for global model fine-tuning in FL. Personalized FL employs fine-tuning to adapt global models to local ones, e.g., FedBABU [38] , FTFA, and RTFA [5] . However, this focus on local performance neglects global generalization. Inspired by linear mode connectivity [36, 12] , Model Soups [52] combines runs with varied hyper-parameters to improving fine-tuning performance in the centralized setting. DiWA [43] and other Soups-based methods [52, 43, 3] extends this concept, emphasizing the importance of model diversity. Some methods induce diversity through high learning rates [34] , cosine similarity minimization [51] , tempered posteriors [20] , or auxiliary dataset-trained model soups [42] . We depict the difference of different model ensemble-based methods in our appendix 7.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper acknowledges several limitations and areas for future research in Sec. 5. Firstly, it highlights a trade-off between training memory and performance when reducing communication rounds in model merging. This indicates an awareness of the potential drawbacks of their method and suggests that further work is needed to make the method more training-memory efficient. Additionally, the paper notes that it focuses exclusively on vision-related tasks, suggesting that extending the findings to language tasks or multimodal scenarios would be a promising direction for future research.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 The structure of this Section is as follows: firstly, we provide the problem definition and corresponding notions to be used (Sec. 3.1); secondly, we reveal the dilemma for existing federated learning methods on reducing communication rounds (Sec. 3.2); finally, we propose a regularized model interpolationbased method as a solution, provide corresponding analysis (Sec. 3.3), and present the overall algorithm flow.Notions. Let X be the input space of data, Y be the label space. Consider a FL setting with M clients, τ local steps and R communication rounds. Let D := {D i } M i=1 be a set of M domain, each of which is a distribution over the input space X . For each client, we have access to n training data points in the form of (X i , Y i ) = {(x i j , y i j )} n j=1 , where y i j denotes the target label for input x i j . Let f ∈ R m represents the parameter for the global model, ℓ i : R m → R denotes the local objective function at client i, and P denotes a distribution on the entire set of clients. We provide a notation table in Appendix A to clarify the meanings of the corresponding notations.Problem definition. We aim to address the challenge of optimizing the global performance on D of aggregated models fine-tuned from different clients with data heterogeneity, while minimizing communication rounds between the clients and the server in data Non-IID setting. In terms of global performance, we perform empirical risk minimization (ERM) on the sampled dataD i for i ∈ [M ], L(f ) = M i=1 p i L i (f ), where L i (f ) = 1 |D i | ξ∈Di ℓ i (f, ξ) and M i=1 p i = 1.(1)In this section, we present a theoretical analysis to understand how communication rounds, local steps, and our introduced regularization terms affect the convergence bound in federated learning. Formally, we present the error term and posit the following assumptions for the purpose of analysis.Our analysis builds on the assumptions and convergence bound in [50] with formal statements in Appendix B.1. We first present a theorem providing a convergence guarantee when the proposed regularization terms are applied. EQUATIONHere, the update rule of the t iteration with the affinity and diversity term is defined as θ(t + 1) = θ(t) -ηg(t) -q(t, µ t , µ a ), and the extra term satisfies q(t, µ t , µ a ) ≤ c. The hyper-parameters µ t and µ a represent the co-efficient of tuning affinity and diversity respectively.Besides, d := ∥f (0,0) -f ⋆ ∥ refers to the distance between initialization f (0,0) and the global optimum f ⋆ , σ bounds variance of stochastic gradient by E[∥g i (f (r,k) ) -∇L i (f (r,k) )∥ 2 |f (r,k) ] ≤ σ 2 ,max i sup f ∇L i (f (r,k) ) -∇L(f (r,k) ) ≤ ζ.How to reduce communication rounds under data heterogeneity? Increasing local fine-tuning steps seems to be a straightforward technique to reduce communication costs. Nevertheless, this approach cannot reduce the an error term in the convergence rate (see the 4th term of the RHS of Eq. 2), which remains unaltered by increasing local steps. Moreover, increasing local update steps in the presence of Non-IID client data exacerbates the inconsistency in local objectives, further magnifying this error term. Here, we provide a more detailed explanation, specifically identifying the conditions under which increasing iteration steps can effectively reduce communication rounds.Proposition 3.2. Under the data heterogeneity setting, when the total number of gradient computations across all clients (K = M τ R) is fixed and the local steps τ satisfiesEQUATIONthe error upper bound Eq.2 will be dominated by the second term O(1/ √ K).We provide the proof for Proposition 3.2 in Appendix B.2. Accordingly, increasing the bound in Eq. 2 and meeting the aforementioned condition for local steps allows us to reduce communication rounds. From the above in-equation, we can observe that although increasing the number of local training steps can reduce communication rounds, there is a limit to the number of steps that can be added. This limit is primarily determined by the error term introduced by local updates. f s ← RandomInterpolation(M) {connectivity preserving} 8: L reg (f pi ) = L(f s , D) + λ a • dist(f pi , f p ) -λ d • dist(f pi , M) 9: f pi ← f pi -η∇ fp i L reg (f pi ) 10:end for 11: end for 12: Inference:13: f ← Averaging(M)Limitation of previous model soups methods. Previous model soups methods [52] can induce a trained model located in a connected low-loss valley, but their training efficiency is exceedingly low, due to two aspects: Time-Consuming model selection phase: Firstly, these methods involve a time-consuming model selection phase, which consists of all candidate models available for weight interpolation [3, 52] . This phase aims to choose models that reside within the same low-loss valleys. During this selection process, significant computational resources are consumed to identify suitable models for interpolation, adding to the overall training time and complexity. Extensive and redundant model training: Secondly, model soups entail an extensive number of model training iterations, lacking prior guidance and relying on brute-force, random, and often redundant training [29, 52] . Many of the trained models end up unused, further exacerbating the computational inefficiency.To address the time-consuming model selection issue of the previous soups-based method, we propose a sequential random model interpolation method during training. This innovative approach streamlines the training process by eliminating the need for subsequent model selection steps within the model pool (i.e., local models to be interpolated), which traditionally consumes a considerable amount of computational resources and time. Let M = {f p1 , f p2 , . . . , f p N } be a pool of N models, where f pi represents the weights of the i-th model. We define the interpolated model f interp as a weighted combination of the models in M. The interpolation coefficients α = (α 1 , α 2 , . . . , α N ) are sampled using a uniform distribution and normalization strategy. The interpolated model f interp is then computed as: f interp = N i=1 α i f pi . Here, α i represents the weight assigned to the i-th model in the pool. The uniform distribution ensures that the coefficients α i are non-negative and sum to 1, providing a simple and effective way to combine the model weights from the pool M. Forward and backward propagation are performed using the interpolated model, updating the weights of the currently active model (i.e., the newly added model ) (corresponding to Algorithm 1 Line 7), while previously added model weights remain fixed.The diversity term is proposed to address the redundant model training issue of the previous soupsbased methods by encouraging low-loss region expansion. In particular, the diversity indicator assesses the variability among models within the model pool by summing the distances between pairs of models. Greater distances between models indicate a higher degree of diversity, which correlates with enhanced model quality. This diversity metric is integrated into the FL local training process as a regularization term to facilitate the extensive enlargement of low-loss regions, consequently maximizing the effectiveness of trained models. The diversity term (in Algorithm 1 Line 8) measures the distance between the current training model and other models that will be averaged, and we hope that this distance to be large. The diversity loss can be defined asEQUATIONHere, fEQUATIONHere, f p is a pre-trained model in the first communication round (R = 1). Moreover, it encourages each local model to lie in a close zone in the parameter space, which is beneficial for subsequent server aggregation, especially under data heterogeneity. We use l 2 distance for the dist(, ) metric for both Eq. 4 and Eq. 5.We outline LSS as follows: We begin with the initialization of the client's local model with the pretrained global model. Then we will refine the local model using affinity and diversity loss. This step is performed for a few local update steps. Finally, after updating local model, we aggregate them in the server following the common averaging operation in FedAvg [35] . The flow of LSS for local updating (Step 2 described in Sec 3.1) can be found in Algorithm 1.In conclusion, our method aims to minimize the distance between the local fine-tuned model and the pre-trained initialized global model while maximizing the distance between the model soups ingredients (i.e., the models to be averaged). Our fine-tuned models find large low-loss regions on their respective local datasets while ensuring parameters close to the pre-trained initialization. It is intuitive that the parameters of our fine-tuned models can be more easily aligned with those of models fine-tuned on similar datasets, thereby improving communication efficiency. [53] and CIFAR10 [26] , while feature shift involves Digit5 and DomainNet. Further information on the specific datasets can be found in the appendix. In the label shift scenario, we partitioned the dataset into five clients and the data for each client are sampled following Dirichlet distributions with coefficient α = 1.0, yielding imbalanced label distributions. In the feature shift scenario, we utilized five clients for Digit5 [13, 30] and five clients for DomainNet [40] . Additional results on an extended number of clients are presented in the appendix.Method Accuracy (R = 1) ↑ Accuracy (R = 3) ↑ Accuracy (R = 1) ↑ Accuracy (R = 3) ↑ FedAvg [Model. In terms of models, we used the ImageNet pre-trained ResNet50 [16] as the base model for the DomainNet dataset, while for other datasets, we used the pre-trained ResNet-18 trained on ImageNet [9] . We also present the experimental results based on the vision transformer (ViT) model [11] with parameter-efficient fine-tuning.Baselines. We compare LSS against the vanilla FL method -FedAvg [35] and several advanced FL algorithms designed for Non-IID settings, including FedProx [28] , MOON [27] , FedBN [30] , FedFomo [58] , FedRep [8] and FedBABU [38] . Additionally, we make comparisons with topperforming weight/model-averaging-based domain generalization methods including SWA [21] , SWAD [1] , Soups [52] and DiWA [43] by adapting them to FL. In particular, the specific approach is to modify the local client training in the FedAvg framework to a corresponding fine-tuning approach.For more details, please refer to the appendix.Evaluation and implementation details. Unless otherwise specified, the model performance in the experiments below refers to the global model performance after aggregation on the server side. Our training optimizer uses the Adam optimizer with a learning rate of 5e-4 and a training batch size of 64. For commonly used FL methods, due to the significant increase in local update steps that leads to worse convergence, we set their local update steps to 8. For SWA, SWAD, and our method, we take more local update steps, with each model being averaged trained 8 steps, and the default number of models to be averaged is 4. For the Model Soups method and DiWA, we train 32 models with 8 steps. Additional details of experiment implementations are included in the Appendix. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: Yes, the paper meticulously outlines all relevant assumptions and presents comprehensive, logically sound proofs, ensuring the validity and reliability of each theoretical result. These elements are included in Sec. 3 and Appendix.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
performance comparison: 
 Results on label shift. To demonstrate the effectiveness of LSS on label shift scenario, we conduct comparison experiments on FMNIST and CIFAR-10 datasets. We consider fine-tuning with an extremely limited number of communication rounds (i.e., R = 1 and R = 3). Table 1 reports the test accuracy with the format of mean (std) for all compared algorithms. All experiments are repeated 3 runs with different random seeds. In Table 1 , LSS achieves the best accuracy on all settings of both datasets, which validates that LSS is efficient and effective in fine-tuning FL for label shift Non-IID.Notably, with just one round of communication, LSS can double the accuracy of the best Non-IID FL baseline method. Surprisingly, the simple extension of model-averaging-based domain generalization methods onto FedAvg [35] (the 2nd big row in Table 1 ) perform very well, especially when the number communication round is small. The superior performance using local weight averaging-based fine-tuning is likely because it significantly reduces the gradient variance of local and global variance (see 3.2). We further provide results on different levels of label shift in the supplementary material.Results on feature shift. Table 2 evaluates on feature shift scenario using Digits-5 and DomainNet datasets. Similar to the previous experiment setting for Table 1 , we repeat all the algorithms with 3 random seeds. Consistent with the observation in Table 2 , LSS is the top-performing method under all the settings for both datasets. We also observe better performance achieved by adapting model-averaging-based domain generalization methods (the 2nd big row in Table 2 ) in FL than the existing Non-IID FL methods (the 1st big row in Table 2 ), which further verifies the effectiveness of model averaging to obtain better global model while improving communication efficiency.Convergence plots. We also evaluate the strength of faster convergence using the proposed LSS compared with FedAvg [35] on CIFAR-10 (label shift) and Digtis-5 (feature shift). Fig. 3 depicts the testing accuracies at early and late phases regarding the number of communication rounds to reach convergence. First, by looking at the final testing accuracies on Fig. 3 Parameter-Efficient Tuning with ViT. We also deployed the Vision Transformer (ViT) [11] in FL learning. On Digits-5 dataset, we evaluate the ViT model with a resolution of 224 and a patch size of 16, which was pretrained on the ImageNet-21k dataset. Due to the large number of parameters in ViT, we used a parameter-efficient fine-tuning method called LoRA [19] to train it for all the methods. For more details about our ViT architecture and LoRA training, please refer to the appendix. It can be observed in Fig. 4 that our method is applicable to pre-trained ViT models, demonstrating that our approach can be combined with parameter-efficient fine-tuning methods to further enhance the communication efficiency of FL.We conducted ablation experiments on the main components (i.e., affinity, diversity term and averaged model quantity) of our proposed method and evaluated their performance on the CIFAR dataset, with the performance metric being the global model performance at communication round R = 1. Investigation on regularization losses. In order to examine the importance of affinity loss and diversity loss, as well as the influence of their corresponding coefficients, we adjust one coefficient within a range of 0 to 4 while maintaining the other at a constant value. By comparing the performance with and without loss term, we observe that adding affinity and diversity terms can enhance the model's performance. Additionally, we observe that the two terms complement each other, and selecting appropriate coefficients can achieve significant performance improvement (e.g., adjusting the affinity coefficient to 3 as shown in Fig. 5 (a) and diversity coefficient to 3 as shown in Fig. 5 (b) ). Investigation on the number of averaged models.To investigate the impact of the averaged model quantity on enhancing communication efficiency and reducing gradient variance between local and global, we experiment with varied model quantities and evaluate their influence on global model performance, averaged local model performance 2 , and worst outof-distribution (OOD) generalization performance on the other clients. Fig. 6 shows that increasing the number of averaged models can improve the model's OOD generalization ability and enhance the performance of the aggregated model. This similar upward trend confirms the validity of our analysis linking OOD generalization and local-global variance. We provide a more detailed analysis on connecting our proposed LSS and OOD generalization in appendix C. Additionally, we can observe that increasing the number of models in our method can improve both pre-aggregation and post-aggregation model performance.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper fully discloses all the information needed to reproduce the main experimental results. This information is provided in Sec. 4, with additional detailed information available in the appendix. 
conclusion: 
 Limitations and Broader Impact. Our method reduces communication rounds but trades off training memory and performance. Future work should explore more memory-efficient deployments. While focused on vision tasks, extending to language and multimodal scenarios is promising. Balancing performance and communication in healthcare FL is promising, but excessive reduction can impair critical medical decisions. Careful trade-off consideration is essential for reliable FL applications in sensitive areas.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully the main experimental results, as described in supplemental material?Answer: [Yes]Justification: Yes, the paper specifies all the training and test details necessary to understand the results, including data splits, hyperparameters, their selection process, and the type of optimizer used. This comprehensive detailing ensures that the experimental setup and outcomes are transparent and reproducible in Section 4.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specifies all the training and test details necessary to understand the results, including data splits, hyperparameters, their selection process, and the type of optimizer used. These details are comprehensively documented in Sec. 4 to ensure clarity and reproducibility.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Yes, the paper reports error bars and other appropriate information about the statistical significance of the experiments in Sec. 4. These details are suitably and correctly defined to ensure the reliability and validity of the reported results.Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: es, the paper discusses both potential positive and negative societal impacts of the work performed. The discussion includes an analysis of how the research can benefit society and also addresses possible adverse effects, ensuring a balanced and comprehensive evaluation of the societal implications in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not describe safeguards for the responsible release of data or models that have a high risk for misuse, such as pretrained language models, image generators, or scraped datasets. Because we does not focus the release of such assets, we focus model training techniques. (see Sec. 4).Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA]Justification: The paper does not utilize external assets such as code, data, or models from other creators or original owners. Therefore, there are no specific credits, licenses, or terms of use that need to be mentioned or respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not introduce any new assets such as code, data, or models. Therefore, there is no documentation provided alongside new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing experiments or research with human subjects. Therefore, it does not include instructions given to participants, screenshots, or details about compensation. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We propose an efficient method, Local Superior Soups (LSS), to reduce communication rounds in FL with pre-trained initialization, addressing the challenge of data heterogeneity. By employing sequential model interpolation, connectivity preservation, and two regularization terms (diversity and affinity), the method allows for an increase in local training steps and a reduction in communication rounds while avoiding client drift. This approach, tailored for pre-trained model adaptation in FL, offers training and inference efficiency, making it suitable for practical deployment in edge computing scenarios. As the first step towards understanding and developing model soupsbased methods in pre-trained models in FL, this study conducts experiments on benchmark datasets. Our method attain superior performance with a only few rounds of communication and surpasses the performance of standard FL methods significantly across four datasets and under two distribution shift scenarios.The appendix is organized as follows. We list the notations table in Section A. We provide the theoretical proof of the convergence analysis in Section B. We present the theoretical intuition of our proposed two loss term C. Next, we provide more detailed related work in Sec. D We present more experiment details and results in Sec. E. Standard FL [35] employs a server to coordinate the following iterative distributed training:Step EQUATION)EQUATIONAssumption B.4. (Bounded regularization update). The update introduced by the affinity and diversity regularization term q(t, µ t , µ a ) in the update rule θ(t + 1) = θ(t) -ηg(t) -q(t, µ t , µ a ), is bounded by a constant c, namely q(t, µ t , µ a ) ≤ c.We have the main theorem on convergence rate, which similar to [49] except for the introduced regularization terms. η = min{ 1 4β , M 1 2 d τ 1 2 R 1 2 , σ , d 2 3 τ 2 3 R 1 3 β 1 3 σ 2 3, dEQUATIONwe defineϵ = E 1 τ R R-1 r=0 τ k=1 β(f (r,k) ) -β(f ⋆ ) , and have ϵ ≤ 2βR 2 τ R + 2σd √ M τ R + 5β 1 3 σ 2 3 d 4 3 τ 1 3 R 2 3+ 15βEQUATIONHere, the update rule of the t iteration with the affinity and diversity term is defined as θ(t + 1) = θ(t) -ηg(t) -q(t, µ t , µ a ), and the extra term satisfies q(t, µ t , µ a ) ≤ c. The hyper-parameters µ t and µ a represent the co-efficient of tuning affinity and diversity respectively.Besides, d := ∥f (0,0) -f ⋆ ∥ refers to the distance between initialization f (0,0) and the global optimum f ⋆ , σ bounds variance of stochastic gradient byE[∥g i (f (r,k) ) -∇L i (f (r,k) )∥ 2 |f (r,k) ] ≤ σ 2 ,max i sup f ∇L i (f (r,k) ) -∇L(f (r,k) ) ≤ ζ.The regularization update bound is reasonable since q(µ t , µ a ) = µ t * (θ -θ m ) -µ a * (θ -θ m ).Here, θ m is the averaged parameter of all the parameter in the model pool for interpolation. As the inherent trade-off effect of diversity and affinity term, q(µ t , µ a ) will not diverge too much in practice.And the the bounded value c can be effectively controlled by tuning hyper-parameter µ a and µ tThe distinguishing factor in our convergence rate, as compared to that in [49] , stems from the unique inter-client update bound facilitated by our proposed regularization term. We have the inter-client (e.g., for client index 1 and 2) update bound asEQUATIONProof. To find a tight bound for ∥∇L 1 (f ) -∇L 2 (f ) -q 1 (µ t , µ a ) + q 2 (µ t , µ a )∥ , we will use the given inequalities:EQUATION)By breaking down the expression with inserting global gradient ∇L and then apply the triangle inequality of absolute value, and our given inequalities, we haveEQUATION)Combined our derived inter-client update bound with the Equation (30) in Appendix D.2 in [49] , we can easily obtain a different bounded client update drift bound ϵ c as follows:EQUATIONLeveraging the above in-equation and choosing the learning rate η properly, we can get our Theorem 3.1.B.2 Proof of Proposition 3.2 Proposition 3.2 Under the data heterogeneity setting, when the total number of gradient computations across all clients (K = M τ R) is fixed and the local steps τ satisfiesEQUATIONthe error upper bound Eq.equation 15 will be dominated by the second term O(1/ √ K).Taking local steps can save total communication rounds compared to synchronous SGD. To be more specific, as suggested in [49] , when the total number of gradient evaluations/computations across all clients (K = M τ R) is fixed and the local steps τ satisfies:EQUATION)When the upper bound of local steps (Eq.( 3)) becomes larger, there will be more communication savings. Therefore, the quantity in Eq.( 3) represents the largest savings in communication rounds. Next, we show the error upper bound under the data heterogeneity setting.Proof. Under high data heterogeneity, we have ζ + c ≥ σ, and:EQUATIONTherefore, we have Proposition 3.2:EQUATIONThis Proposition 3.2 indicates that when client data are Non-IID, the side effects of the error term in the Theorem 1 will be further exacerbated, therefore, increasing the local iteration steps effectively reduces the communication rounds.C Theoretical Intuitions.Connecting ζ with out-of-distribution error. Ensemble is a category of the promising method that ensembles trained models to improve generalizability as demonstrated in centralized settings via reducing model discrepancy [21] . To reduce the variance ζ of local and global gradients that is resulted by data heterogeneity, we aim to adapt ensemble to FL. Intuitively, local client training that can reduce the error on the worst domain (client) in FL will reduce the variance ζ.In the following, we detail how to reduce ζ with OOD error with a bias-variance-covariance-locality (BVCL) decomposition analysis. ensemble can be defined as:f WA ≜ 1/N N n=1 f n .We have the following decomposition of ensemble's expected test error. Bias-variance-covariance-locality decomposition. The expected generalization error on domain T of f WA over the joint distribution (L N S ≜ {l(N ) S } N N =1) of N learning procedure on domain S is:EQUATIONHere, cov refers to the covariance of predictions made by two member models. The first component is the same bias as that of each individual member. The variance of ensemble is split into two parts: the variance of each member divided by the number of members (N ) and a covariance term. The last locality term enforces constraints on the weights to ensure the functional ensembling approximation remains valid. In summary, combining N models reduces variance by a factor of N , but introduces the covariance and locality terms which must be controlled to ensure low OOD error.In the analysis presented in [43] , the authors proposed a BVCL decomposition based on the approximation of functional ensembling (i.e., averaged prediction instead of parameter) by WA. The expected generalization error on domain T of f WA over the joint distribution (L N S ≜ {l(N ) S } N N =1) of N learning procedure on domain S is:E L N S E T (f WA (L N S )) = E (x,y)∼p T bias 2 (x, y) + 1 N var(x) + N -1 N cov(x) + O( ∆2 ), (BVCL)Definition C.1 (Bias). For x ∈ X and y ∈ Y , we define the bias of OOD prediction as,EQUATIONDefinition C.2 (Variance). For x ∈ X, we define the variance of prediction asEQUATION)Definition C.3 (Covariance). For x ∈ X, we define the covariance of prediction produced by two different learning procedures l S and l ′ S as covEQUATIONDefinition C.4 (Locality). For any averaged models f i (for i ∈ [N ]), i is the index of an averaged model, N is the total number of averaged models, we define the locality of all averaged models asEQUATIONFollowing the definitions of the terms in the BCVL generalization bound, we discuss the insights of reducing the bound via the proposed strategy. Our method is based on WAFT, which enjoys the benefit of reducing prediction variance by averaging the predictions of multiple models. The diversity term in our proposed method reduces the covariance term by encouraging functional diversity in the parameter space. The affinity term in our proposed method reduces the locality term to ensure the approximation of weight averaging (WA) to prediction ensembling.Analysis on variance. One can see that an increase in the number of averaged models can directly lead to a reduction in variance. The straightforward averaging M models, as seen in the vanilla WAFT method, diminishes variance by a factor of M . However, this approach also introduces covariance and locality terms, which necessitate meticulous management on adding new averaged models to guarantee minimal out-of-distribution (OOD) error.Analysis on covariance. The covariance term represents the predictive covariance between two member models whose weights are averaged. It increases when the predictions of different averaged models are highly correlated. In the worst-case scenario where all predictions are identical, the covariance is equal to the variance, rendering the benefits of weight averaging ineffective [43] . Conversely, when the covariance is lower, the advantages of weight averaging over individual models become more pronounced. Therefore, it is crucial to address covariance by promoting functional diversity among the averaged models. Our proposed method incorporates a diversity term that aims to reduce this covariance.Analysis on locality. The locality term, which represents the expected squared maximum distance between weights and their average, constrains the weights to be close and ensures the approximation. The affinity term in our proposed method encourages the reduction of this locality term.Overall, to reduce WA's error in OOD, we need to seek a good trade-off between diversity and locality. Our solution achieves this balance through two optimizable loss terms, the diversity term, and the affinity term. Besides, the direct combination of M models, as in the vanilla WAFT method, reducesSuperior Soups variance by a factor of M but introduces covariance and locality terms that need to be carefully managed in order to ensure low OOD error.Inference Training Single Final Model Curated IngredientsIt is worth noting that, from an implementation perspective, unlike the model soups method (see Fig. 7 middle), which requires retraining a large number of candidate models for model selection and interpolation, our method only selects a few models (typically 3 to 5) for sequential random interpolation training in order to maintain connectivity. This significantly reduces the time cost of local training. Furthermore, unlike model ensembles (see Fig. 7 ) that require storing multiple model weights and integrating predictions during inference, our method only needs to retain an averaged weight during the final inference stage. This greatly reduces the memory footprint and enhances the inference speed on the client side.FL performance downgrading on Non-IID data is a critical challenge. A variety of FL algorithms have been proposed to handle this heterogeneous issue. From an optimization perspective: FedProx [28] adds L 2 norm to the client model and the previous server model to regularize them. This helps to prevent the client models from diverging too far from the server model. Scaffold [25] adds a variance reduction term to mitigate the "clients-drift." MOON [27] uses mode-level contrastive learning to stabilize local training by making the client models more robust to changes in the data distribution. In addition, personalized FL [46] is another approach to achieving high local testing performance on Non-IID data. For aggregation perspective: FedBN [30] uses local batch normalization to alleviate the feature shift before averaging models. For extreme communication efficient: In recent years, there have been some FL methods based on one-shot communication rounds. These methods typically use additional techniques on the server-side, such as using prediction ensembles [14] instead of weight ensembles or generating data [56, 18] from local models for centralized training, to improve the performance of the aggregated model. These methods are orthogonal to our client training-based approach. There are also works on few-round communication rounds in FL based on meta-learning frameworks [39] , but the data partition used in the experimental setup may not be suitable for practical FL scenarios.Fine-tuning aims to achieve improved performance on the given task by leveraging the learned knowledge of the pre-trained model. [7] empirically study the impact of fine-tuning from a pretrained model in FL and unsurprisingly find that starting from a pre-trained model reduces the training time required to reach a target error rate and enables the training of more accurate models than starting from random initialization. [57] propose a knowledge distillation approach for fine-tuning the global model, called FedFTG. In addition, fine-tuning in FL has been widely used in personalized FL to address Non-IID problems by having each user adapt the global model to personalized local models using their own data. For example, FedBABU [38] splits the model into body and head, then fine-tuning the head part for personalization. [5] propose FTFA and RTFA that start with a pre-trained model and then fine-tunes a small subset of model parameters using the FedAvg [35] algorithm. However, this line of work focuses on optimizing local performance and ignores the generalization of global data. This can lead to a performance drop when we further update the global model from the updated local models. Weight averaging and model recycling are not only efficient ways to aggregate machine learning models but also present promising benefits of improving model generalizability. Inspired by the linear mode connectivity property of neural networks trained with stochastic gradient descent (SGD) [36, 12] , Model Soups [52] proposes to combine many independent runs with varied hyper-parameter configurations. Similarly, DiWA [43] utilizes this idea of Model Soups while theoretically analyzing the importance of training different models with diverse hyper-parameters within mild ranges. Soups-based methods [52, 43] rely on aggregating diverse models to improve model generalizability. To induce greater diversity, some methods such as [34] using a high constant learning rate, [51] minimizing cosine similarity between weights, [20] using a tempered posterior and model Ratatouille [42] averages diverse model trained from auxiliary datasets.Dataset. We validate the effectiveness of our proposed method with four datasets, FMNIST [53] , CIFAR-10 [26], Digit-5 [13, 30] , and DomainNet [40] . The Fashion-MNIST (FMNIST) dataset is a dataset of Zalando's article images consisting of a training set of 60, 000 examples and a test set of 10, 000 examples. Each example is a 28 × 28 grayscale image of a piece of clothing. The dataset is divided into 10 classes: t-shirt/top, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, and ankle boot. The CIFAR-10 dataset is a popular dataset for machine learning research. It consists of 60, 000 32 × 32 color images divided into 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. The dataset is split into 50, 000 training images and 10, 000 test images. The Digit-5 dataset is a collection of five popular digit datasets, MNIST [10] (55000 samples), MNIST-M (55000 samples), Synthetic Digits [13] (25000 samples), SVHN (73257 samples), and USPS (7438 samples). Each digit dataset includes a different style of 0-9 digit images. The DomainNet dataset is a large-scale dataset of images collected from six different domains: clipart, infograph, painting, quickdraw, real, and sketch. The dataset contains 600, 000 images, each labeled with one of 345 object categories. The images in the DomainNet dataset are of high quality and are diverse in terms of their content and style.Model. We used the pre-trained models from the timm repo1 , which are a collection of state-of-theart deep learning models for computer vision tasks. For our proposed LSS, we use Adam optimizer with a learning rate of 5e-4 , momentum 0.9, and weight decay 5e-4. The default number of averaged models is 4. Each model updates 8 epoch then aggregates with the others. The default affinity term coefficient is 3 and diversity term coefficient is 3. We set the batch size to 64 by default. For vision transformer (ViT) [11] Training Details. We implement all the methods in PyTorch, and we run all the experiments on an NVIDIA Tesla V100 GPU. Unless otherwise specified, the model performance in the experiments below refers to the global model performance after aggregation on the server side. For commonly used FL methods, due to the significant increase in local update steps that leads to worse convergence, we set their local update steps to 8.Applying WAFT to FL Local Update. For SWA [21] , SWAD [1] , and our method LSS, we take more local update steps, with each model being averaged trained 8 steps, and the default number of models to be averaged is 4. For the Model Soups [52] method and DiWA [43] , we trained 32 models and each model trained 8 steps. The hyper-parameter configuration for model selection includes learning rate ([1e-4, 5e-4, 1e-5]), batch size ( [32, 64, 128] ), dropout rate ([0.0, 0.1, 0.3]), and weight decay [5e-4, 5e-5, 5e-6]. Each run randomly select one of the hyper-parameter options. From each run of WAFT method, we take the weights of the epoch with maximum accuracy on the validation dataset, which follows the training distribution.Arbitrarily increasing local steps cannot reduce communication rounds.From Table 4 , we can see that simply increasing local steps does not always lead to improved model performance. For FedAvg on the CIFAR10 dataset, increasing local steps beyond 8 actually results in a decrease in model performance. Computational and memory costs comparison.In Table 5 , we provide detailed information on computational overhead and memory usage for various methods. Since the computational overhead and memory usage of FedAvg and other used FL methods are nearly identical, we only present the data for FedAvg here. Similarly, as the computational overhead and memory usage for SWA and SWAD, as well as for Soups and DiWA, are also nearly the same, we only show the data for SWA and Soups methods. It can be observed that our method requires more memory compared to other soups-based methods. However, the overall computational time for a single client's communication round is faster in our approach. This is because other soups-based methods require training a large number of models repeatedly to achieve good model performance. For instance, Soups needs to train 32 models, whereas our method only requires training 4 models. If the number of models trained by Soups is reduced to just 4, it only brings about a 5% improvement compared to FedAvg with a communication round of 1. LSS encourages smoothness (reducing β). In Table 6 , we provide the performance degradation of trained models evaluating under varying levels of random noise. Generally, a smaller performance degradation indicates a more robust model, which to some extent reflects the smoothness of the trained model. We can observe that our method exhibits greater robustness to noise perturbation. LSS improves flatness of loss landscape. The sharpness measure utilized in the Table 7 computes the median of the dominant Hessian eigenvalue across all training set batches through the Power Iteration algorithm [54] . This metric signifies the maximum curvature of the loss landscape, commonly employed in the literature on flat minima [23] to indicate sharpness. As demonstrated in the presented table, it is clear that our proposed method results in flatter minima compared to FedAvg. Evaluation with more clients. To assess the effectiveness of our method in larger-scale client scenarios, we conducted an expanded experiment involving 50 clients. From the Table 8 , we can observe that our proposed method maintains a significant advantage across different client scales, particularly when the number of communication rounds is small (R = 1). FedAvg and LSS. Our method, LSS, when applied to large language models for instruction tuning, achieves higher scores than the common FedAvg. This suggests that LSS is a promising approach for improving performance and convergence in federated learning settings for large language models, in addition to its success in image classification.Method Accuracy (R = 1) ↑ Accuracy (R = 3) ↑ Accuracy (R = 1) ↑ Accuracy (R = 3) ↑ FedAvg [Exploring the use of our method in a diverse set of complex LLM tasks is an interesting direction for future research.Evaluation with ViT. To validate the effectiveness of our method across different network architectures, we conducted an expanded experiment using the Vision Transformer (ViT) model based on the Transformer architecture. Upon observing the Table 9 , it is evident that our method consistently enhances the communication efficiency of federated learning with ViT model architectures.Evaluation with different Non-IID level. To further comprehensively validate the effectiveness of our method under different levels of data heterogeneity, we conducted experiments on the CIFAR-10 dataset by adjusting the coefficients α of the Dirichlet distribution. We examined the performance of our method in scenarios with greater distribution variations. Based on the Table 10 , it is evident that our method maintains a significant advantage in scenarios with larger data heterogeneity.Evaluation with different Initialized Models. To compare the performance of our method under different types of parameter initialization, we conducted experiments on the CIFAR-10 dataset using both pretrained and random initialization. Table 11 shows that our method still maintains a significant advantage with random initialization, but it does not achieve the near-optimal performance seen with pre-trained initialization. Comparison of Convergence Speed Between FedProx and LSS. Fig. 9 shows the accuracy of the testing during the early and late phases in terms of the number of communication rounds required to reach convergence. These results demonstrate that our method outperforms FedProx in both the early and late phases of federated learning.Method Accuracy (R = 1) ↑ Accuracy (R = 3) ↑ Accuracy (R = 1) ↑ Accuracy (R = 3) ↑ FedAvg [Setup. We follow the setup of Fed-Aya [55] , which involves four iterative steps: server-to-client model downloading, local model training, client-to-server model uploading, and global model aggregation. For instruction tuning, we use the parameter-efficient fine-tuning technique, LoRA [19] , applied to the Llama-7b model.Dataset. We use the Aya dataset [44] , a multilingual instruction tuning dataset with annotations from contributors worldwide. Our experiments include 6 high-resource languages (English, Spanish, French, Russian, Portuguese, Chinese) and 2 low-resource languages (standard Arabic, Telugu). The dataset is filtered to include contributors with at least 100 annotations, resulting in 38 clients with a total of 25k data samples.Model. The model used for our experiments is the Llama2-7b [48] , fine-tuned using the LoRA technique. We evaluate the effectiveness of the training methods using an in-domain evaluation metric termed Ref-GPT4 [60] , where GPT-4o rates the generated responses against ground-truth responses. The score given by GPT-Ref ranges from 0 to 10. We adopt the same prompt template used in FedLLM-Bench [55] . The implementation of applying our method to LoRA is the same as that used in the ViT experiments (see Fig. 4 ) described earlier.Result. Our method, LSS, when applied to large language models for instruction tuning, achieves higher scores than the common FedAvg. This suggests that LSS is a promising approach for improving performance and convergence in federated learning settings for large language models, in addition to its success in image classification. Exploring the use of our method in a diverse set of complex LLM tasks is an interesting direction for future research.Table 11 : Different model initialization (Pre-trained v.s. Random): Label shift test accuracy after R = 1 and R = 3 communication rounds. Result: It shows that our method still maintains a significant advantage with random initialization, but it does not achieve the near-optimal performance seen with pre-trained initialization.CIFAR-10 (Pre-trained) CIFAR-10 (Random) Guidelines:Method Accuracy (R = 1) ↑ Accuracy (R = 3) ↑ Accuracy (R = 1) ↑ Accuracy (R = 3) ↑ FedAvg [• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.the average performance of local models of individual clients before aggregation on the overall client datasethttps://github.com/huggingface/pytorch-image-models 


Decision: All three reviewers have provided positive feedback on this submission. The paper introduces a novel method to reduce communication costs in federated learning for pre-trained models. The authors have adequately addressed the concerns and weaknesses raised by the reviewers. Based on this, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes LSS, a model interpolation-based local training technique to reduce the number of communication rounds required. The intuition is to regularize local models to connected low-loss valleys, so the aggregated model may have lower loss. LSS is empirically evaluated on a variety of datasets and types of distribution shifts.
strength_weakness: strength_weakness: - Readability: the notation in section 3 is not clear enough. For example, $n$ is used for both number of data (section 3.1) and number of averaged model (Alg 1), which is confusing. 
- (minor) It might be a little abuse of notation when using $\mathcal{D}_i$ for both distribution and dataset. I suggest using different notations.
questions: questions: - Although Figure 2 is intuitive, it might not be so rigorous: high affinity and high diversity do not necessarily guarantee an aggregated model with low loss. In the example, two client’s local models are different “horizontally”, while the major axis of the loss landscape is also horizontal. However, this is not always guaranteed. If the clients models are different “vertically”, the proposed method may just fail. Mathematically, in algorithm 1, the gradient of $dist(f_{p_i}, f_p)$ and $dist(f_{p_i}, \mathcal{M})$ is very likely to be nearly orthogonal. Could you explain intuitively why LSS is expected to perform well in most of the cases? 
- The scope of this paper is limited to FL with a pre-trained model. I understand that the pre-trained model may be larger in average, which makes convergence more changing. I am curious how LSS is limited to pre-trained models and whether it can generalize to randomly initialized models.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a method called Local Superior Soups (LSS), a novel technique for model merging in cross-silo federated learning aimed at reducing communication rounds while enhancing model performance. This paper introduces random interpolation, diversity term, and affinity term to alleviate the need for time-consuming model selection and redundant model training. Rigorous experiments on 4 datasets with 11baselines demonstrate the effectiveness of LSS.
strength_weakness: strength_weakness: 1. The distinction between LSS and similar federated learning methods such as FedProx, which also incorporates weights from global models to regulate client loss, is not clearly discussed.
2. The subsection 3.3.1 titled "Random interpolation conserving connected low-loss region." lacks mathematical detail to fully understand the interpolation process.
3. The requirement for clients to receive the interpolated model pool ($M$) could potentially lead to significant communication overheads, which may not present a clear advantage over simpler methods like FedAvg or FedProx.
4. The connection between Theorem 3.1 and the core methodology of LSS, specifically the diversity and affinity terms, appears tenuous. These terms do not seem to be directly derived from the theorem, which may weaken the theoretical foundation of the proposed method.
5. This paper should consider referencing relevant literatures or conducting preliminary experiments to support its statments on the part called "Limitation of previous model soups methods".
questions: questions: 1. In Figure 3, how do the results of LSS compare with those of FedProx? Given the similarities in the core concepts between LSS and FedProx, such a comparison would be insightful for evaluating the distinct advantages of LSS.
2. The results presented in Tables 1 and 2 suggest that LSS performs well in the initial training rounds. Can the authors clarify whether LSS is primarily advantageous only during these initial rounds? 
3. Furthermore, how should the training process be continued post-initialization? Is it feasible to employ LSS throughout the entirety of the training process, or would alternative methods be more effective in later stages?
limitations: limitations: Please see the weaknesses and the questions.
paper_summary: paper_summary: In this paper Local Superior Soups (LSS) is proposed to minimize communication rounds in federated learning (FL) using pre-trained models, specifically tackling data heterogeneity challenges. LSS achieves this by employing sequential model interpolation, maintaining connectivity, and integrating diversity and affinity regularization terms. These innovations enable more local training steps and fewer communication rounds, effectively preventing client drift. Designed for adapting pre-trained models in FL, LSS enhances training efficiency, making it well-suited for deployment in edge computing applications.
strength_weakness: strength_weakness: 1. Only two small-scale image datasets are used in experiments. More large-scale datasets, especially those in other modalities, should be used.

2. More pretrained models should be explored.

3. More tasks besides image classification should be incorporated into experiments.
questions: questions: None.
limitations: limitations: See weakness part

Comments: title: Appreciation for the Positive Feedback and Score Increase
Thank you for your answers.
comment: We sincerely appreciate the time and effort you have invested in reviewing our work and providing such valuable feedback. We hope that our rebuttal has thoroughly addressed your comments and concerns.

As the author-reviewer discussion period nears its conclusion on August 13th, we would like to kindly invite you to contact us if there are any additional questions or points that you believe require further discussion. Your insights have been crucial in refining our submission, and we remain fully committed to addressing any remaining issues.

If you find that our responses and improvements adequately address your concerns, we would be deeply grateful if you could consider raising your score. Thank you once again for your thoughtful review.
We sincerely appreciate the time and effort you have invested in reviewing our work and providing such valuable feedback. We hope that our rebuttal has thoroughly addressed your comments and concerns.

As the author-reviewer discussion period nears its conclusion on August 13th, we would like to kindly invite you to contact us if there are any additional questions or points that you believe require further discussion. Your insights have been crucial in refining our submission, and we remain fully committed to addressing any remaining issues.

If you find that our responses and improvements adequately address your concerns, we would be deeply grateful if you could consider raising your score. Thank you once again for your thoughtful review.
We are truly grateful that our responses have successfully addressed all of your concerns. Thank you for your positive feedback and for raising our score. We deeply appreciate your valuable comments and the time you’ve taken to review our work!
Thank you for your answers. After reading your clarifications, I have raised my score, and I have no other concerns. I appreciate your effort.