Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning
Authors: Minghui Chen
Meirui Jiang
Xin Zhang
Qi Dou
Zehua Wang
Xiaoxiao Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning; Model Merging
Primary area: other

Abstract: Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. 
Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. 
However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. 
To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.''
Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. 
This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL.
We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets.

Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning
Minghui Chen
Meirui Jiang"
Xin Zhang
Qi Dou
Zehua Wang'
Xiaoxiao Li "Meta
University of British Columbia
Vector Institute
SChnnese University of Hong Kong
Abstract
Federated learning (FL.) is a learning paradigm that enables collaborative training of models using decentralized data. Recently, the utilization of pre-trained weight ini- tialization in FL has been demonstrated to effectively improve model performance. However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. To address these communication cost issues and increase the performance of pre-trained model adaptation FL, we propose an innovative model interpolation- ing technique called "Local Superior Soups." Our method enhances local training across different clients, encouraging the exploration within few communication rounds through regularized model interpolation. This approach acts as catalyst for the seamless adaptation of pre-trained models in FL. We demonstrated its effectiveness and efficiency across diverse widely-used FI. datasets. Our code available at https://github.come wttpgghtwww.ooouutttttaaaaa.aa.e wwwl--upeeiio-SSSSS Soups.
local train-
connected low-loss basin
Introduction
Federated learning (FI.) (35] has emerged promising methodology for leveraging the power of private data without the need for centralized data governance. However, data heterogeneity in FL poses significant challenges the design of efficient training for global convergence. With the emergence of the pre-training and fine-tuning paradigm in various applications [15] 19]. recent studies 2 have attempted address the problem of FL under data heterogeneity with pre-trained initialization. Although pre- pre-trained federated learning can speed up convergence compared to random initialization, it still requires significant number of communication rounds between the server and clients, often amounting to hundreds of rounds [37]. Existing pre- traitraine models 41 47 often have an enormous parameter scale, and following neural scaling law (24), there is continuous trend toward increasing model parameters. Deploying models with such large parameter size in FL. introduces significant communication overhead. This greatly hampers the flexibility and scalability of model updates. Reducing FL communication overhead can be approached by reducing the scale of model parameters involved in distributed training [59] or reducing communication rounds [35]. Comparing with reducing model parameters, reducing communication rounds typically leads to a more efficient reduction of network congestion [17), decreased energy consumption on client devices [33] and lower risk of privacy breaches (61). In this paper, we focus reducing communication rounds in FL with pre- pre-trained model as inirialization.
Typically, increasing the number of local training steps can effectively reduce communication rounds. However, there is an upper limit to the extent of local training step increments. This limitation arises due to the presence of data heterogeneity, where the consistency of optimization among different clients deteriorates with the increasing number of local steps [35]. This optimiza- tion inconsistency leads to discrepancy between local and global models and decelerates the
Corresponddncce to xiaoxiao.... wwweee.eeeeee. uilecce.ucc.ca
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

convergence rate of FL. The discrepancy is often called client drift 125 Previously, some FL methods [25] 45 attempted to introduce proximal terms regularize local training, with the aim of reducing local overfitting and minimizing the problem of client drift. While these methods can accelerate convergence, they restrict the progress of each local training steps towards the optimal solution, impeding the attainment of FL, with more aggressive communication round reductions. While these client drift mitigation methods can reduce local overfitting some extent, they can- not ensure strong performance of the global aggregated models, particularly in scenarios with limited communication rounds. This sit- uation arises when individual local clients be- come trapped in isolated low-loss valleys. More specifically, as illustrated in Figure T two mod- els from clients 'A' and 'B', even their optimal model distance is small, still result in a poorly performing aggregated global model. Moreover, the preceding FL methods aimed at minimizing communication rounds exclusively address scenarios involving random initialization, lacking a customized approach tailored to pre-trained models. Recent a greedy model selection version of model soups) based on model interpolation (averages of a large number of model weights) are effective approaches to seek large connected low-loss region, which are promising for applying in FL. to reduce communication rounds. These methods can prevent individual clients from being trapped in isolated low-loss valleys by positioning the global model centrally within a larger low-loss region by overlapping the low-loss regions among clients, as shown in Fig. [ (right). However, their training efficiency is exceedingly low, requiring complete retraining of numerous models, leading to significant computational overhead on clients and intolerable communication costs when applied in FL, due to two aspects: First, they involve a time-consuming model selection phase within the model pool, which consists of all candidate models available for weight interpolation. Secondly, model soups entail an extensive number of model training iterations, lacking prior guidance and relying on brute-force, random, and often redundant training. Many of the trained models end up unused.
Isolated Low-Loss Valley
Connected Low-Loss Valley
Low Loss
Client Model
Client Model
Global Model
Global
Model
Illustration on isolated (left) and connected low-loss valley with larger regions in dark red (right).
Figure 1:
proposed centralized one-tuning methods (e.g., model soups [52 and DiWA 43
To enjoy the connected low-loss valley benefits of model soup-based methods 52 without burdening local training, we propose an efficient and local model interpolation-base method, called Local Superior Soups (LSS). To address the first issue, we propose sequential ran- dom model interpolation method during training. This eliminates the need for subsequent model selection steps and ensures that the models slated for interpolation reside within the same low-loss valley during training (Sec. B.3.T). For the second issue, we introduce two quantifi- able indicators of candidate model quality, inspired by data augmentation quality quantifica- tion [32 4 diversity (Sec. (33.2) and affinity (Sec. 3.3.3p. Specifically, the diversity indica- tor quantifies the diversity among models in the model pool with their pairwise model distance, where larger distances denote higher diversity, signifying better model quality for interpolation. As illustrated in Figure 2
(left), low-loss re- gion, supported by models with low diversity, can be effectively covered with only a few candi- date models positioned near its periphery. Thus, propose incorporating the diversity metric as a regularization term during training to maxi- mize the expansion of low-loss regions, thereby increasing the utilization of trained models. The affinity indicator measures the affinity of each candidate model in the model pool to the ini- tial model. Smaller distances indicate greater affinity, indicating better model quality for in- terpolation. This affinity is also incorporated as regularization term during training to prevent the expansion of low-loss regions from deviating too far from the shared initialization point, thus increasing the likelihood of overlapping connected
Diversity
Affinity
small covered region
Tong distance from Init.
small overiapping region
High Diversity
High Affinity
Inht Modiel
Medel
High
large covered ragion
from
large overlapping region
Figure 2: Illustration on diversity (left) and affinity (right) regularization.
regions (as depicted on the right side of Fig. 2 These two indicators facilitate the efficient inclusion of models into the model pool, preventing wasteful training of models that may ultimately go unused.

---

In experiments, we found that our proposed method greatly reduces communication rounds, and we achieved the performance of models fused after multiple rounds of communication in other FL methods with only a few rounds of communication.
In summary, contributions
follows.
(I) We reveal the importance of regularizing local client models in the connected low-loss valleys for reducing communication rounds when initializing FL with pre-trained models. (2) We introduce an innovative and efficient model soups-based method for FL, called Local Superior Soups (LSS) that eliminates the need for time-consuming model selection and redundant model training in the existing soups-based approaches, while expanding connected low-loss valleys of client models for faster convergence. (3) In experimental evaluations, LSS demonstrates significant reduction in communication rounds, achieving superior performance with only few rounds of communication, exceeding baseline FI. methods significantly in four datasets and two types of distribution shifts.
2 Related Work
2.1 Heterogeneous Federated Learning
FL struggles with Non-IID data, leading various proposed algorithms. FedProx 28
uses proximal
term to regularize local training, preventing client divergence. Scaffold [25] adds variance reduction to combat "clients-drift. MOON [27] employs mode-level contrastive learning to stabilize local training.
Personalized FL [46] targets high local performance on Non-IID data. FedBN [30] applies local batch
normalization mitigate feature shift before model averaging. Recent one-shot and few-round FL
methods use parallel server-side techniques like prediction ensembles [14], data generation [56]
or meta-learning 339 to improve aggregated model performance.
2.2 Federated Fine-tuning and Model Interpolation
line-tuning leverages pre-trained models to enhance task performance 77. FedFTG [57] proposes knowledge distillation for global model fine-tuning in FL. Personalized FL employs fine-tuning to adapt global models local ones, e.g.. FedBABU [38, FTFA, and RTFA 3 However, this focus on local performance neglects global generalization. Inspired by linear mode connectivity [36 12).
Model Soups [52] combines runs with varied hyper-parameter to improving fine-tuning performance in the centralized setting. DiWA 43 and other Soups-based methods [52] 43 3 extends this concept, emphasizing the importance of model diversity. Some methods induce diversity through high learning rates (34), cosine similarity minimization [51), tempered posteriors (20). or auxiliary dataset-trained
model soups [42]. We depict the difference of different model masemble-basee methods our
appendix
3 Method
The structure of this Section is as follows: firstly, we provide the problem definition and corresponding notions to be used (Sec.B.Il) secondly, we reveal the dilemma for existing federated learning methods on reducing communication rounds (Sec. 3.2); finally, we propose regularized model interpolation- based method as a solution, provide corresponding analysis (Sec. 3.3). and present the overall algorithm flow.
3.1 Notions and Problem Definition
Let X' be the input space of data, y be the label space. Consider FL setting with M clients, local steps and communication rounds. Let == { be set of M domain, each of which is distribution over the input space X. For each client, we have access to training data points the form of (X, J1) = { ,})) where y denotes the target label for input æ/. Let E R.m represents the parameter for the global model, f; RT + IR denotes the local objective function client and P denotes distribution on the entire set of clients. We provide a notation table in Appendix A to clarify the meanings of the corresponding notations.
Notions.

---

Problem definition. We aim to address the challenge of optimizing the global performance on D of aggregated models fine-tuned from different clients with data heterogeneity, while minimizing communication rounds between the clients and the server in data Non-IID) setting. In terms of global performance, we perform empirical risk minimization (ERM) on the sampled data Dj for E [M].
M C(f) = Σ p.L,(),, where L,(f) = D
M
Σ EED;
4(f.C) and _
P: =1
3.2 Effect of Regularization and Local Steps on FL Convergence under Data Heterogeneity
In this section, we present theoretical analysis to understand how communication rounds, local steps, and our introduced regularization terms affect the convergence bound in federated learning. Formally. we present the eтrоr term and posit the following assumptions for the purpose of analysis. Our analysis builds on the assumptions and convergence bound in [50] with formal statements in Appendix B.I] We first present theorem providing a convergence guarantee when the proposed regularization terms are applied.
Theorem 3.1 (Convergence Rate for Convex Local Functions with Affinity and Diversity Constraint). Under Convexity and Smoothness Assumption on 3-smooth loss function, Bounded Variance of Stochastic Gradient and Bounded Variance of Local and Global Gradient assumptions, when the
Mtd.
43
43
client learning rate is chosen properly as, 1 min 13 FiRa, =3R50303 35533
we define e = E
RR-
1
B (1))) 3( -
and have
T
23R
2nd
58toid
15 +)
+
TR
MrR R3
RS
Here, the update rule of the t iteration with the affinity and diversity term is defined as 0(t + 8(t) - ng(() qt, Pes Ma), and the extra term satisfies gtt, fr, Ma) I c. The hyper-parameters f4 and Ne represent the co-efficient of tuning affinity and diversity respectively.
Besides, d = - fI refers to the distance between initialization f(0.0) and the global optimum
f a bounds variance of stochastic gradient by (()))))(()))) - (( < -,, and bounds variance of local and global gradient by max; sup, (L.(f((A))) - (e(((n))) || S
How to reduce communication rounds under data heterogeneity? Increasing local fine-tuning steps seems to be a straightforward technique to reduce communication costs. Nevertheless, this approach cannot reduce the an error term in the convergence rate (see the 4th term of the RHS of 2q. which remains unaltered by increasing local steps. Moreover, increasing local update steps in the presence of Non-IID client data exacerbates the inconsistency in local objectives, further magnifying this error term. Here, we provide a more detailed explanation, specifically identifying the conditions under which increasing iteration steps effectively reduce communication rounds.
Proposition 3.2. Under the data helerogeneity setting, when the total number of gradient compula- tions across all clients (K- = Mr R) is fixed and the local steps satisfies
σ Kt
0
TS
(3)
d3
M2
the upper bound 2q will be dominated by the second term O1/VKK).
We provide the proof for Proposition 3.2 in Appendix B.2 Accordingly, increasing the bound in Eq. [ and meeting the aforementioned condition for local steps allows us to reduce communication rounds. From the above in-equation, we can observe that although increasing the number of local training steps can reduce communication rounds, there a limit to the number of steps that can be added. This limit is primarily determined by the error term introduced by local updates.

---

Why connecting low-loss valley in local training with pre-trained initialization can achieve extreme communication rounds reduction? Our analysis indicates that for substantial communica- tion efficiency in federated learning, it is not enough to just increase local training steps. The focus should be on minimizing the error term from local updates, particularly the last term in Formula 2 This term, influenced by gradient dissimilarity ((), distance to optimal weights (d), and our proposed regularization update bound C, remains significant even as training steps increase.
Prior research suggests 37 that pre-training initialization reduces ( by aligning client updates, and overparameterrizatio ensures that the optimal parameters are typically close to the initialization [22] 6 B11, decreasing d. It is important to note that the regularization related term e is influenced by a combination of model diversity and affinity, and can be reduced by adjusting the parameters fe and Pa- In the absence of a common pre-trained initialization, ensuring model affinity within the model pool is often challenging (i.e., models from different clients tend diverge significantly from the initialization values), resulting in a larger value of c, which in turn affects the effectiveness of our method. Therefore, our approach is more suitable when combined with pre-trained models. Consequently, a combination of pre-training and our proposed connectivity preserving local training can effectively lower error terms from local updates, increasing the limit of local training steps and thus reducing communication rounds. More experimental support see our Appendix.
3.3 Our Solution: LSS Algorithm
In this part, we first present the shortcomings of the previous model soups method applied in FL. Secondly, we propose our three targeted improvements, ie. random model interpolation (Sec. (.3.1. diversity term (See. (.3.2), and affinity regularization term (Sec. 3.3.3). Finally, we present the complete algorithm process and detailed implementation in local client training.
Limitation of previous model soups methods. Previous model soups methods [52] can induce a trained model located in a connected low-loss valley, but their training efficiency is exceed- ingly low, due to two aspects: Time-Consiin Consuminn model selection phase: Firstly, these methods in- volve a time-consuming model selection phase, which consists of all candidate models avail- able for weight interpolation (5)) This phase aims to choose models that reside within the same low-loss valleys. During this selection process, significant computational resources are consumed to identify suitable models for inter- polation, adding to the overall training time and complexity. Extensive and redundant model training: Secondly, model soups entail exten- sive number of model training iterations, lack- ing prior guidance and relying random, and often redundant training 22 [2]. Many of the trained models end up unused, fur- ther exacerbating the computational inefficiency.
Algorithm 1 LSS (Local Training) Pseudo-code
Require:
fp pre-trained model (R = 1) or global aggregated model (R > 1); L loss function; 'D dataset; dist distance function; T iteration steps; 1) learning rate; Xa affinity coefficient; Xa diversity coefficient; number of averaged models.
1: LSS Local Training:
2: M t {fp}
3: for p; to N do
4:
fp. + Averaging(M)
5:
M <- M-M (s::) (sequential training with
newly added model)
6:
for
to r do
7:
fa f Random Interollaioon
(connectivity preserving]
forceforce,
8:
Lreg
L(Sa, D) + Xa
dist(f fr) - >d fp.t(
9:
fp. + fp. n/ sr. Freg (p.))
10: end for
11: end for
3.3.1 Random interpolation conserving connected low-loss region.
12: Inference:
13: f Averaging(M)
To address the time-consuming model selection issue of the previous bases method, we propose a sequential random model interpolation method during training. This innovative approach streamlines the training process by eliminating the need for subsequent model selection steps within the model pool (i.e., local models to be interpolated), which traditionally consumes considerable amount of computational resources and time. Let M = (pp .ps be pool of N models, where fp. represents the weights of the i-th model. We define the interpolated model fimeer weighted combination of the models in M. The interpolation coefficients a = (ar, (22 ... aN)) ) are sampled using a uniform distribution and normalization strategy. The interpolated model fineop is
5

---

then computed as: finter= = -E in the pool. The uniform distribution ensures that the coefficients a; are non-negative and sum to 1, providing a simple and effective way to combine the model weights from the pool M. Forward and backward propagation are performed using the interpolated model, updating the weights of the currently active model (i.e., the newly added model ) (corresponding to Algorithm[]] D Line 7), while previously added model weights remain fixed.
asp.. Here, a; represents the weight assigned to the i-th model
3.3.2 Diversity term.
The diversity term is proposed to address the redundant model training issue of the previous soups- based methods by encouraging low-loss region expansion. In particular, the diversity indicator assesses the variability among models within the model pool by summing the distances between pairs of models. Greater distances between models indicate a higher degree of diversity, which correlates with enhanced model quality. This diversity metric is integrated into the FL local training process as a regularization term to facilitate the extensive enlargement of low-loss regions, consequently maximizing the effectiveness of trained models. The diversity term (in Algorithm[]] Line 8) measures the distance between the current training model and other models that will be averaged, and we hope that this distance to be large. The diversity loss can be defined
N dist(J, Jn).
1 Ediversiry = dis((I, = N Σ
Here, fn belongs to local interpolated model pool M and N is the number of local candidate models. The candidate models (i.e., model soups ingredients) are models to be interpolated in local training, and the model pool the set of local candidate models (see Algorithm| [ Line 5). We use the lg norm to measure the distance between model weights.
3.3.3 Affinity term.
The affinity term is proposed to control the expansion of low-loss regions and prevent local candidate model training divergence. The affinity indicator assesses the level of alignment between each candidate model within the model pool and the initial global model by calculating the cumulative distances between each candidate model and the initialization model. Smaller distances between models signify a stronger affinity, indicating higher model quality. To ensure the controlled expansion of low-loss regions and reduce the probability of overlapping connected regions, this affinity metric is integrated into the training process as a regularization term. The affinity term (in Algorithm [Line 8) measures the distance between the candidate model and the initial model weights, with the aim of minimizing this dissimilarity (maximize this loss term) to ensure that the distance remains relatively small. The affinity loss can be defined
latfnity dist(f fp).
(5)
Here, fp is a pre-trained model in the first communication round (R = 1). Moreover, it encourages each local model to lie in close zone in the parameter space, which is beneficial for subsequent server aggregation, especially under data heterogeneity. We use l2 distance for the dist(,) ) metric for both Eq. 4 and Eq. 5
3.3.4 Overall pipeline.
We outline LSS as follows: We begin with the initialization of the client's local model with the pretrained global model. Then we will refine the local model using affinity and diversity loss. This step is performed for few local update steps. Finally, after updating local model, aggregate them in the server following the common averaging operation in FedAvg [35). The flow LSS for local updating (Step 2 described in Sec 3.1 can be found in Algorithm
In conclusion, our method aims to minimize the distance between the local fine-tuned model and the pre-trained initialized global model while maximizing the distance between the model soups ingredients (i.e., the models to averaged). Our fine-tuned models find large low-loss regions on their respective local datasets while ensuring parameters close to the pre-trained initialization. It intuitive that the parameters of our fine-tuned models can be more easily aligned with those of models fine-tuned on similar datasets, thereby improving communication efficiency.

---

Table Label shift test accuracy after and It = communication rounds. We primarily compared two
categories of methods: conventional FL methods and state-of-the-ar local weight averaging- -aaee fine-tunne tie-tuning methods that enhance domain generalization.
FMNIST
CIFAR-10
Method
Accuracy (R 1)
Accuracy (R 3)
Accuracy (Rt 1
Accuracy (R 3)
FedAvg 351
35.34(1.71)
(..32))
58.34(0.86)
66.74(0.76)
FedProx 28
33.48(1.52)
89.28
56. 56.74(0.92)
63.21
MOON 27
36.01(1.66)
91.28(0.33)
58 58.96
67.04(1.12)
FedBN 30
34.20(1.73)
09.87(0.47)
67.04(0.75)
64.51(0.677
FedForno 588
33.94(1.65)
88.41(0.69)
55.01(0.89)
62.69(0.75)
FedRep 88
36.20(1.52)
91.07(0.23)
57.73
66.23(0.73)
FedBABU 38
36.18(1.43)
(0.31(
60.14(1.06)
67.16(0.87)
SWA 21
55.82(1.02)
91.03(0.11)
59.07(1.228
67.45(1.15)
SWAD 1
58 (0).87)
91.22(00.66
60 60.54(1.15)
(7.65(0.97)
Soups
(0.11)
91.56(0.24)
61.00(1.04)
67.63(0.94)
DiWA [
(3.24)
01.88(0.13)
61.32(1.26)
68.05(1.10)
LSS Mem.)
(2.66
92.45 (0.21)
65.96,
75.16(L07
Table 2: Feature shift test accuracy after A 1 and
communication rounds. LSS consistently outperforms other methods on both datasets across under feature shift settings.
Digit-5
DomainNet
Method
Accuracy (R 1)
Accuracy (H 3)
Accuracy (ft
1)
Accuracy (R 3)
FedAvg 35
46.36(2.08)
80.48(0.81)
18.76(3.52)
29.43(2.01)
FedProx 28
41.01(1.92)
77.83
17.27
27.18(2.29)
MOON 27
50.11(1.72)
83.02(0.64)
19.61(3.54)
31.27(2.34)
FcdBN
80
46.02(1.93)
81.42
18.16(3.09)
28.65(1.89
FedFomo
58
41.87(2.13)
76.21
15.10(3.82)
25.69
FedRep 8
47.43(1.73)
(..63)
18.89(2.60)
30.42( 1.84)
FedBABU
38
(8.02(1.81)
(3.79
19.44(2.43)
32.06( 1.88)
SWA [21
54.12(
85.33 62)
22.07(2.55)
35.90 1.611
SWAD 1
57 (7.02(0.71)
86.84(0.64)
21.98(2.61)
36.73(1.57)
Soups
(9.71(0.82)
87.07
22.75(2.85)
38.02(1.00)
DIWA
[
(1.53(0.88)
(8.69
24.88
38.32(1.50)
LSS Mem.)
(2.86(1.64)
(2.97(
27.86
41.35(L.46)
Experiment
4.1 Experimental Setup
Dataset. Our experimental section considers two scenarios of Non-lID settings, namely label shift and feature shift. The label shift scenario investigates datasets such as FMNIST (53) and CIFARIO 10 (26.. while feature shift involves Digit5 and DomainNet. Further information on the specific datasets be found in the appendix. In the label shift scenario, partitioned the dataset into five clients and the data for each client are sampled following Dirichlet distributions with coefficient = 1.0, yielding imbalanced label distributions. In the feature shift scenario, we utilized five clients for Digit5 133 30 and five clients for DomainNet [4)]. Additional results on an extended number of clients are presented in the appendix.
Model. In terms of models, we used the ImageNet pre-trained ResNet50 16 as the base model for the DomainNet dataset, while for other datasets, used the pre-trained ResNet-18 18 trained on ImageNet (9) We also present the experimental results based on the vision transformer (ViT) model 11 with parameter-efficient fine-tuning.
Baselines. We compare LSS against the vanilla FL method - FedAvg (35 and several advanced
FL algorithms designed for Non-IID settings, including FedProx (28). MOON 27), FedBN (30), FedFomo [58], FedRep
S and FedBABU 338. Additionally, we make comparisons with top-
based domain generalization methods including SWA [2I]. SWAD [. Soups [52] and DIWA [43] by adapting them to FL. In particular, the specific approach is
performing weigh//model-averee
modify the local client training in the FedAvg framework corresponding fine-tuning approach.
For more details, please refer to the appendix.
Evaluation and implementation details. Unless otherwise specified, the model performance in the experiments below refers to the global model performance after aggregation on the server side. Our training optimizer uses the Adam optimizer with a learning rate of 5e-4 and a training batch size of
7

---

64. For commonly used FL methods, due to the sigmificant increase in local update steps that leads to worse convergence, we set their local update steps to 8. For SWA, SWAD, and our method, we take more local update steps, with each model being averaged trained 8 steps, and the default number of models to be averaged For the Model Soups method and DiWA, we train 32 models with 8 steps. Additional details of experiment implementations are included in the Appendix.
CeA Phans)
144
Digit (Barll Piaso!
123
120
(a)
(b)
(c)
(d)
Figure 3: Convergence comparison our proposed LSS with FedAvg. LSS achieves high accuracy much earlier
(around 6 to 8 rounds) than FedAvg, which takes hundreds of communication rounds.
4.2 Performance Comparison
Results on label shift. To demonstrate the effectiveness of LSS on label shift scenario, we conduct comparison experiments on FMNIST and CIFAR-10 datasets. We consider fine-tuning with an extremely limited number of communication rounds (i.e., R and R = 3). Table reports the test accuracy with the format of mean (std) for all compared algorithms. All experiments are repeated 3 runs with different random seeds. In Table I LSS achieves the best accuracy on all settings of both datasets, which validates that LSS is efficient and effective in fine-tuning FL. for label shift Non-IID. Notably, with just one round of communication, LSS can double the accuracy of the best Non-IID FL baseline method. Surprisingly, the simple extension of model- -anddel-vergiig---ee based domain generalization methods onto FedAvg 335 (the 2nd big row in Table I perform very well, especially when the number communication round is small. The superior performance using local weight wveraging-bssed fine-tuning is likely because significantly reduces the gradient variance of local and global variance (see 3.2b.
We further provide results on different levels of label shift in the supplementary material.
Results on feature shift. Table evaluates on feature shift scenario using Digits-5 and DomainNet datasets. Similar to the previous experiment setting for Table we repeat all the algorithms with 3 random seeds. Consistent with the observation Table 2 LSS is the top-performing method under all the settings for both datasets. We also observe better performance achieved by adapting model-averaginggbaae based domain generalization methods (the 2nd big row in Table 2 in FL than the existing Non-IID FL methods (the 1st big row in Table 2 which further verifies the effectiveness of model averaging to obtain better global model while improving communication efficiency.
Convergence plots. We also evaluate the strength of faster convergence using the proposed LSS compared with FedAvg (35) on CIFAR-10 (label shift) and Digtis-5 5 (feature shift). Fig. depicts the testing accuracies at early and late phases regarding the number of communication rounds to reach convergence. First, by looking at the final testing accuracies on Fig. 3 (b) and (d), LSS achieves better performance. Second, Fig. 3 (a) and (c) show that LSS almost meets the targeted performance at the very early stage (i.e.around 6 to rounds), whereas FedAvg requests over hundreds of communication rounds.
Parameter-Effice Tuning with VIT. We also de- ployed the Vision Transformer (ViT) [1] in FL learn- ing. On Digits-5 dataset, we evaluate the VIT model with a resolution of 224 and a patch size of 16, which was pretrained on the ImageNet-21k dataset. Due to the large number of parameters in ViT, we used a parameter-officient fine-tuning method called LoRA [19 to train it for all the methods. For more details about our ViT architecture and LoRA training. please refer to the appendix. It can be observed in Fig. 4 that our method is applicable to pre-trained ViT models, demonstrating that our approach can be combined with parameter-efficient fine-tuning methods to further enhance the communication efficiency of FL.
Communication Round:
Communication Round:
$323
N2.75
Festeo
Paodoo
Parti
Fatter
go
Figure 4: Evaluation on VIT fine-tuned with LoRA (Digit5 dataset).
8

---

4.3 Ablation Studies
We conducted ablation experiments on the main components (že., affinity, diversity term and averaged model quantity) of our proposed method and evaluated their performance the CIFAR dataset, with the performance metric being the global model performance at communication round R 1.
Investigation on regularization losses. In order to examine the importance of affinity loss and diversity loss, as well as the influence of their correspond- ing coefficients, we adjust one coefficient within a range of 0 to 4 while maintaining the other at con- stant value. By comparing the performance with and without loss term, we observe that adding affinity and diversity terms can enhance the model's perfor- mance. Additionally, complement each other, and selecting appropriate co- Figure 5: Ablation on the affinity & diversity.
Altilty Verke
Divariity Varke
61-11
ARfoor
20.0
24.
00
03.0
(a)
(b)
observe that the two terms
efficients can achieve significant performance improvement (e.g., adjusting the affinity coefficient to 3 as shown in Fig. S
(a) and diversity coefficient to 3 as shown in Fig. (b)).
Investigation on the number of averaged models. To investigate the impact of the averaged model quan- tity enhancing communication efficiency and re- ducing gradient variance between local and global, experiment with varied model quantities and eval- uate their influence on global model performance, averaged local model performance and worst out- of-distribution (OOD) generalization performance on the other clients. Fig. 6 shows that increasing the number of averaged models can improve the model's s OOD generalization ability and enhance the perfor- mance of the aggregated model. This similar upward trend confirms the validity of our analysis linking OOD generalization and local-global variance. We provide more detailed analysis on connecting our proposed LSS and OOD generalization in appendix [ Additionally, we can observe that increas- ing the number of models in our method can improve both pre-aggregation and post-aggregation model performance.
(b)
Figure 6: Ablation studies on the impact of the number of averaged models communication efficiency and performance variance. We evaluated the influence of varied model quantitics on global and averaged local model performance, as well as generalization on the worst client.
Conclusion
Limitations and Broader Impact. Our method reduces communication rounds but trades off training memory and performance. Future work should explore more memory-efficient deploy- ments. While focused on vision tasks, extending to language and multimodal scenarios is promising. Balancing performance and communication in healthcare FL is promising, but excessive reduction can impair critical medical decisions. Careful trade-off consideration is essential for reliable FL applications in sensitive areas.
Conclusion. We propose an efficient method, Local Superior Soups (LSS), to reduce communication rounds in FL with pre-trained initialization, addressing the challenge of data heterogeneity. By employing sequential model interpolation, connectivity preservation, and two regularization terms (diversity and affinity), the method allows for increase in local training steps and reduction in communication rounds while avoiding client drift. This approach, tailored for pre-trained model adaptation FL, offers training and inference efficiency, making it suitable for practical deployment in edge computing scenarios. As the first step towards understanding and developing model soups- based methods in pre-trained models in FL, this study conducts experiments on benchmark datasets. Our method attain superior performance with only few rounds of communication and surpasses the performance of standard FL methods significantly across four datasets and under two distribution shift scenarios.
"te average performance of local models of individual clients before aggregation on the overall client dataset

---

Acknowledgement. M. Chen, Z. Wang and X. Li are grateful for the support of the Natural Science and Engineering Research Council of Canada (NSERC). M. Chen and X. Li are supported by the Canada CIFAR AT Chairs program, MITACS -FFCC--CIFAR Catalyst Grant Program, NVIDIA Hardware Awards, the Digital Research Alliance of Canada, and Canada Foundation for Innovation (CFI). M. Jiang and Q. Dou are supported by the Research Grants Council of the Hong Kong Special Administrative Region (Project No. T45- 401/22-N).
References
[I] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: domain generalization by seeking flat minima. In NewrIPS, pages 22405-22444 2021.
[2] Hong-You You Chen, Cheng-Hao Tu, Ziwei Li, Han-Wei Shen, and Wei-Lun Chao. On the impor- tance and applicability of pre-training for federated learning, 2022.
[3] Minghui Chen, Meirui Jiang, Qi Dou, Zehua Wang, and Xiaoxiao Li. Fedsoup: Improving generalization and personalization in federated learning via selective model interpolation. In MICCAI (2), volume 14221 of Lecture Notes in Computer Science, pages 318-328 Springer, 2023.
[4] Minghui Chen, Cheng Wen, Feng Zheng, Fengxiang He, and Ling Shao. VITA: A multi-source vicinal transfer augmentation method for out-of-distribution generalization. In AAAI, pages 321-329, AAAI Press, 2022.
[5] Gary Cheng, Karan N. Chadha, and John C. Duchi. Fine-tuning is fine in federated learning. CoRR, abs/2108.07313, 2021.
[6] Lénaïc Chizat, Edouard Oyallon, and Francis R. Bach. On lazy training in differentiable programming. In NeurlPS, pages 2933-2943, 2019.
[7] Leshem Choshen, Elad Venezian, Shachar Don- Yehiya, Noam Slonim, and Yoav Katz. Where to start? analyzing the potential value of intermediate models. CoRR, abss2211.000 1.00107, 2022.
[8] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 2089-2099, PMLR, 2021.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pages 248-255. IEEE Computer Society, 2009.
[10] Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag., 141-)11 2012.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[12] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 3259 3269. PMLR, 2020.
[13] Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML,, volume 37 of JMLR Workshop and Conference Proceedings, pages 1180-1189. JoLR.org, 2015.
[14] Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated Icarning. CoRR, abs/1902.1 175, 2019.
[15] Kaiming He, Ross B. Girshick, and Piotr Dollár, Rethinking imagenet pre-training. In ICCV, pages 4917-4926. IEEE, 2019.
10

---

[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778. IEEE Computer Society, 2016.
[17] Parikshit Hegde, Gustavo de Veciana, and Aryan Mokhtari. Network adaptive federated learning: Congestion and lossy compression. In INFOCOM, pages 1-10. IEEE, 2023.
[18] Clare Elizabeth Heinbaugh, Emilio AuczRicca, and Huajie Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.
[19] Edward J.
Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen- Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net, 2022.
[20] Pavel Izmailov, Wesley J. Maddox, Polina Kirichenko, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Subspace inference for bayesian deep learning. In UAI, volume 115 of Proceedings of Machine Learning Research, pages 1169-1179 AUAI Press, 2019.
[21] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. UAI, pages 876-885, AUAI Press, 2018.
[22] Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS, pages 8580-8589, 2018.
[23] Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt Kusner. Questions for flat-minima optimization of modern neural networks. CoRR, abs/2202.00661, 2022.
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Sealing laws for neural language models. CoRR, 014/2001.08366 2020.
[25] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In /CML, volume 119 of Proceedings of Machine Learning Research, pages 5132- 5143. PMLR, 2020.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. corr, 2009.
[27] Qinbin Li, Bingsheng He, and Dawn Song. Modellcontrastive federated learning. In CVPR, pages 10713-10722. Computer Vision Foundation IEEE, 2021.
[28] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In MLSys misys.org. 2020.
[29] Weishi Li, Yong Peng. Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey, CoRR, abs/2309.11668, 2023.
[30] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning non-iid features via local batch normalization. In ICLR. OpenReview.net, 2021.
[31] Xinyan Li and Arindam Banerjee. Experiments with rich regime training for deep learning. CORR, abs/2102. 13522, 2021.
[32] Raphael Gontijo Lopes, Sylvia J. Smullin, Ekin D. Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. CoRR, abs/2002.08973. 2020.
[33] Bing Luo, Xiang Li, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Cost-effective federated learning in mobile edge networks. IEEE J. Sel. Areas Commun., 39(12)3600-3361, 2021.
[34] Wesley J. Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In NeurIPS, pages 13132-13143, 2019.

---

[35] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication -ffficient learning of deep networks from decentralized data. In AISTATS. volume 54 of Proceedings of Machine Learning Research, pages 1273-1282. PMLR, 2017.
[36] Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. In NeurlPS, pages 11611-11622, 2019.
[37] John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to be- gin? on the impact of pre-training and initialization in federated learning. CoRR, abs/2210.08090, 2022.
[38] Jaehoon Oh, Sangmook Kim, and Se- Young Yun. Fedbabu: Towards enhanced representation for federated image classification. In ICLR. OpenReview.net. 2022.
[39] Younghyun Park, Dong-Jun Han, YooY Kim, Jun Seo, and Jackyun Moon. Few-round learning for federated learning. In NeurIPS, pages 28612-28622, 2021.
[40] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, pages 1406-1415, IEEE, 2019.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pages
PMLR, 2021.
[42] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez- Paz. Recycling diverse models for nut-of-distribution generalization. CoRR, 10s/2222.11045, 2022.
[43] Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy. Patrick Galli- nari, and Matthieu Cord. Diverse weight averaging for distt--iitti generalization. In NeurlPS, 2022.
[44] Shivalika Singh, Freddie Vargus. Daniel D' D'souza, Börje F. Karlsson, Abinaya Mahendiran, Wei- Ko, Herumb Shandilya. Jay Patel, Deividas Mataciunas, Laura O' O'Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaci, Irem Ergün, Ifcoma Okoh, Aisha Alaagib, Oshan Mudannayake, 7aid Alyafeai, Minh Chien Vu, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. CoRR, abs/2402.06619, 2024.
[45] Yan Sun, Li Shen, Tiansheng Huang, Liang Ding, and Dacheng Tao. Fedspeed: Larger local in- terval, less communication round, and higher generalization accuracy. In ICLR. OpenReview.net, 2023.
[46] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. CoRR, abs/2103.00710, 2021.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Ferree, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta,
12

---

Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307. 0092888. 2023.
[49] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al- Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917, 2021.
[50] Jianyu Wang. Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Agtiera
y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas N. Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He. Samuel Horváth. Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz. Satyen Kale, Sai Praneeth Karimireddy, Jakub Konečný, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi, Sashank J. Reddi, Peter Richtárik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake E. Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang. Tong Zhang, Chunxiang Zheng, Chen 2021.
Zhu, and Wennan Zhu. A field guide to federated optimization. CoRR,
[51] Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 1217-11227. PMLR, 2021.
[52] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 23965-23998. PMLR, 2022.
[53] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
[54] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. Pyhessian: Neural networks through the lens of the hessian. In IEEE BigData, pages 581-590. IEEE, 2020.
[55] Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen. Fedllm-bench: Realistic benchmarks for federated learning of large language models. CoRR, abs/2406.008855, 2024.
[56] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. DENSE: data-free one-shot federated learning. In NeurlPS, 2022.
[57] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In CVPR, pages 10164-10173. IEEE, 2022.
[58] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In ICLR. OvenRevie 2021.
[59] Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficien tuning methods of pre- trained language models. In ACL (Findings), pages 9963-9977 Association for Computational Linguistics, 2023.
[60] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging alm-as-a-judge with menhe and chatbot arena. In NeurlPS, 2023.
[61] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In NeurlPS, pages 14747-14756, 2019.
13

---

Roadmap of Appendix The appendix is organized as follows. We list the notations table in Section A We provide the theoretical proof of the convergence analysis in Section We present the theoretical intuition of our proposed two loss term [ Next, we provide more detailed related work in Sec. D We present more experiment details and results in Sec.
A Notation Table
Table Important notations used in the paper.
Notations
Description
model parameters
learning procedure
feature dimension
m
number of samples a sample label
set of training domain loss function number of clients number of averaged models total communication rounds
M
N
R
input space of data
Y
label space
coeff. for local training reg. term
local training steps
B Convergence Analysis
B.1 Formal Restatement of Convergence Theorem
Standard FL
355 employs a server to coordinate the following iterative distributed training:
Step In each global round of training [R]. the server broadcasts its current global model
weight f <(r-1) to all the clients;
Step The selected client e copies the current server model weight frr + fg. performs 7 local
step updates, then sends for - 1(r-1) back to the server;
Step 3 The server aggregates the updates from all clients ( - S (r-11) )== to form the new server
model using the weighted averaging Eq
Note that the initialization f(0.0) ,ithh the subscription indicating model at 0-th communication round and 0-th local step, is a pre-trained model (e.g. using public datasets) in our problem. This work focus on improving Step to explore a larger low-loss region local clients.
Formally, we present the convergence results (Theorem 3.T and specify the following formal assumptions: 1) Convexity and Smoothness Assumption B-smooth loss function, 2) Bounded Variance of Stochastic Gradient Assumption and 3) Bounded Variance of Local and Global Gradient Assumption).
Assumption B.1. (Convexity and Smoothness). C; is convex and f-smooth for all E [M],
((,(w)-(L((() < B|[ - =u],
for all w. in its domain and i [M].
Assumption B.2. (Bounded variance of stochastic gradient). Each client can achieve unbiased stochastic gradient with x² -unuioormly bounded variance for all I: E [0, r), namely
Elg. (r. (((()) (C.(
[f (r.kk)
(c.( (11))(1)))))))))
(r.0)
s 5 (6)
14

---

Assumption B.3. (Bounded variance of local and global gradient). The difference of local gradient (f, and the global gradient Vf(f) is bounded in lg norm, namely
max sup VL.U, - sc
(7)
Assumption B.4. (Bounded regularization update). The update introduced by the affinity and diversity regularization term qt, Pt: Ma) in the update rule 0(t + 1) = O(t) ng(() - q(t, Mts Mo), is bounded by a constant c, namely
g(t, Mt, Ma) S c.
(8)
We have the main theorem on convergence rate, which similar to regularization terms.
49 except for the introduced
Theorem 3.1: Convergence Rate for Convex Local Functions with Affinity and Diversity Constraint Under Convexity and Smoothness Assumption on 8-smooth loss function, Bounded Variance of Stochastic Gradient and Bounded Variance of Local and Global Gradient assumptions, when the client learning rate is chosen properly as,
1 Mid
dà
d
mmmnn
(9)
43 7
T T
TR B (C + c)
3(f)) -
we define = E TR D Ek=
and have
23R² 6 E
2nd
33
+ (C
RR +
VMTR
r
R
(10)
Here, the update rule of the iteration with the affinity and diversity term is defined as A(t + 1) @(t) - ng(() qtt, Pt, Ma). and the extra term satisfies g(t, Hr, (a,, S c. The hyper-parameters Pec and Na. represent the co-efficient of tuning affinity and diversity respectively.
Besides, d = [ f*| refers to the distance between initialization f(0.0) and the global optimum
f", bounds variance of stochastic gradient by B[[lg, (f(.t))))))))))))))) (r.(((( ((r))))))))))))))))))))))))))))) I o², and
bounds variance of local and global gradient by max, sup, VL.(f((nk)) V[(f(rk)
I|| S.
The regularization update bound is reasonable since q(k: Haa) μ4 (0-0m) pm. Ha (0 - 6m). Here, 6m is the averaged parameter of all the parameter the model pool for interpolation. As the inherent trade-off effect of diversity and affinity term, q(Ate, (aa) will not diverge too much in practice. And the the bounded value can be effectively controlled by tuning hyper- rrper-parameter μı and μt
The distinguishing factor in our convergence rate, as compared to that in 449 stems from the unique inter-client update bound facilitated by our proposed regularization term. We have the inter-client (e.g., for client index 1 and 2) update bound as
(F) - VL2(f) (u Ma) + 92(14, & 4(5² + 2çc)
(11)
Proof. To find tight bound for (f) vL2(f) - (p + (e(Me,Hall we will use the given inequalities: VC, (L(((r.)))))))) S.
VC VL()))I (IV2(()) VC()I SC.
(12)
By breaking down the expression with inserting global gradient VL and then apply the triangle inequality of absolute value, and our given inequalities, we have
[V - V22()) - ②i (Pt: pa) + 92(pt, pa)l < (2C + 2c)
(13)
Combined our derived inter-client update bound with the Equation (30) in Appendix D.2 in 49 we can casily obtain different bounded client update drift bound as follows:
c, S 47m² 0² + 14 c)²
(14)
Leveraging the above in-equation and choosing the learning rate n properly, we can get our Theo- rem 3.1
15

---

B.2 Proof of Proposition 3.2
Proposition 3.2 Under the data heterogeneity setting, when the total number of gradient computations across all clients (K .MRR) is fixed and the local steps 7 satisfies
K d3 M
(15)
C +
the error upper bound Eq. Eqq.quuaiion 15 will be dominated by the second term O(1/VK).
Taking local steps can save total communication rounds compared to synchromous SGD). To be more
specific, as suggested in [49], when the total number of gradient evaluatiion/ccompuaaiins across all
clients (K = .M is fixed and the local steps satisfies:
K d3 M C+cl
K d3 M
S min
(16)
When the upper bound of local steps (Eq. (3) becomes larger, there will be more communication
savings. Therefore, the quantity in (3.(3) represents the largest savings in communication rounds.
Next, we show the error upper bound under the data heterogeneity setting.
Proof. Under high data heterogeneity, we have Ç + c 2 o, and:
K d3 M
K
K
1 c+c
S
(17)
d3 M
d3 M
Therefore, have Proposition 3.2:
K
0
S
(18)
++c
dB M
This Proposition 3.2 indicates that when client data are Non-IID, the side effects the error term in the Theorem will be further exacerbated, therefore, increasing the local iteration steps effectively reduces the communication rounds.
C Theoretical Intuitions.
C.1 Decomposition of Generalization Bound
Connecting ( with out-of-distribution error. Ensemble a category of the promising method that ensembles trained models improve generalizability demonstrated in centralized settings via reducing model discrepancy [21]. To reduce the variance Ç of local and global gradients that is resulted by data heterogeneity, aim adapt ensemble FL. Intuitively, local client training that can reduce the error on the worst domain (client) in FL will reduce the variance Ç.
In the following, detail how reduce with OOD error with (BVCL) decomposition analysis. ensemble can be defined as: fWA & 1/N M. fn- We have the following decomposition of ensemble's is expected test error. hilu-varrianc--ovarinnee decomposition. The expected generalization error domain T of fwa over the joint distribution
(T.) A (1()) )
of learning procedure on domain is:
N-1 N
N Egg Ex (fwA (L = E( bias" (x, v) + var(x) +
cov(()) + O(A),,
(19)
Here, cov refers to the covariance of predictions made by two member models. The first component is the same bias as that of each individual member. The variance of ensemble is split into two parts:
16

---

the variance of each member divided by the number of members (N) and a covariance term. The last locality term enforees constraints on the weights ensure the functional ensembling approximation remains valid. In summary, combining N models reduces variance by factor of N, but introduces the covariance and locality terms which must be controlled to ensure low OOD error.
In the analysis presented in [43, the authors proposed a BVCL decomposition based on the ap- proximation of functional ensembling (i.e., averaged prediction instead of parameter) by WA. The expected generalization error on domain of fwA over the joint distribution (I- - (1S )N=1) of N learning procedure on domain S is:
Bug EnyyET( (T(WA = B(z,))<p biass(r, (x, y) + var(x) N cov(()) + +(A2), (BVCL) N
N
Definition C.1 (Bias). For a E X and y E Y. we define the bias of OOD prediction as, bias(r, y) = y - Egg (xx,,
(20)
Definition C.2 (Variance). For X, define the variance of prediction as
var(s) Es (J(x, Itg [fx,, [s))²
(21)
Definition C.3 (Covariance). For X, define the covariance of prediction produced by two different learning procedures Is and I's as
cov(x) = Big.1, (((xx, Esg (s(x,ss))) (f(x, - EEg (f(x, (s)))
(22)
Definition C.4 (Locality). For any averaged models fi (for E [N], i is the index of an averaged model, is the total number of averaged models, we define the locality of all averaged models as
A = Egg A2 with Any = =max [I - fwall2-
NN
(23)
x=1
Following the definitions of the terms in the BCVL generalization bound, we discuss the insights of reducing the bound via the proposed strategy. Our method is based WAFT, which enjoys the benefit of reducing prediction variance by averaging the predictions of multiple models. The diversity term in proposed method reduces the covariance term by encouraging functional diversity in the parameter space. The affinity term in our proposed method reduces the locality term to ensure the approximation of weight averaging (WA) prediction ensembling.
Analysis on variance. One can see that an increase in the number of averaged models can directly lead reduction in variance. The straightforward averaging M models, seen in the vanilla WAFT method, diminishes variance by factor of M. However, this approach also introduces covariance and locality terms, which necessitate meticulous management on adding new averaged models to guarantee minimal out-of-distribution (OOD) error.
Analysis on covariance. The covariance term represents the predictive covariance between two member models whose weights are averaged. It increases when the predictions of different averaged models are highly correlated. covariance is equal to the variance, rendering the benefits of weight averaging ineffective [4]] Conversely, when the covariance lower, the advantages of weight averaging over individual models become more pronounced. Therefore, it is crucial to address covariance by promoting functional diversity among the averaged models. Our proposed method incorporates a diversity term that aims to reduce this covariance.
the worst-case scenario where all predictions are identical, the
Analysis on locality. The locality term, which represents the expected squared maximum distance between weights and their average, constrains the weights to be close and ensures the approximation. The affinity term in our proposed method encourages the reduction of this locality term.
Overall, to reduce WA's emor in OOD, we need to seek good trade-off between diversity and locality, Our solution achieves this balance through two optimizable loss terms, the diversity term, and the affinity term. Besides, the direct combination of M models, in the vamilla WAFT method, reduces
17

---

Model Ensemble
Model Soups
Superior Soups
Curated
Ingredients
Training
Single Final Model
Inference
Figure 7: Comparison model ensemble, model soups, and superior soups.
variance by factor of M but introduces covariance and locality terms that need to be carefully managed in order to ensure low OOD error.
worth noting that, from an implementation perspective, unlike the model soups method (sce Fig. middle), which requires retraining large number of candidate models for model selection and interpolation, our method only selects a few models (typically to 5) for sequential random interpolation training in order to maintain connectivity. This significantly reduces the time cost of local training. Furthermore, unlike model ensembles (see Fig. 7 that require storing multiple model weights and integrating predictions during inference, method only needs retain averaged weight during the final inference stage. This greatly reduces the memory footprint and enhances the inference speed on the client side.
D
More Related Work
D.1 Heterogencous Federated Learning
FL performance downgrading on Non-IID data is critical challenge. variety of FL algorithms have been proposed to handle this heterogeneous issue. From an optimization perspective: FedProx [28] adds Le norm the client model and the previous prevent the client models from diverging too far from the server model. Scaffold (25) adds variance reduction to mitigate the "clients-drift" MOON [27] uses mode-level contrastive learning to stabilize local training by making the client models more robust to changes in the data distribution. In addition, personalized FI. [46] another approach achieving high local testing performance on Non-III) data. For aggregation perspective: FedBN [30 uses local batch normalization to alleviate the feature shift before averaging models. For extreme communication efficient: recent years, there have been some FL methods based on one-shot communication rounds. These methods typically use additional techniques on the server-side, such as using prediction ensembles [14] instead weight ensembles generating data [56]]] 18] from local models for centralized training, to improve the performance of the aggregated model. These methods are orthogonal to our client training-bases approach. There also works few-round communication rounds in FL based on meta-learning frameworks [39], but the data partition used in the experimental setup may not suitable for practical FL scenarios.
model regularize them. This helps to
D.2 Federated Fine-tuning and Model Interpolation
Fine-tuning aims achieve improved performance on the given task by leveraging the learned knowledge of the pre-trained model. 7 empirically study the impact of fine-tuning from a pre- trained model in FI. and unsurprisingly find that starting from a pre-trained model reduces the training time required to reach target error rate and enables the training of more accurate models than
18

---

starting from random initialization. [57] propose knowledge distillation approach for fine-tuning the global model, called FedFTG. In addition, fine-tuning in FL has been widely used in personalized FL. to address Non-IID problems by having each user adapt the global model to personalized local
models using their own data. For example, FedBABU (38 splits the model into body and head, then
fine-tuning the head part for personalization. [5] propose FTFA and RTFA that start with a pre-trained
model and then fine-tunes a small subset of model parameters using the FedAvg [35] algorithm. However, this line of work focuses optimizing local performance and ignores the generalization of global data. This can lead to performance drop when we further update the global model from the updated local models. Weight averaging and model recycling machine learning models but also present promising benefits of improving model generalizability. Inspired by the linear mode connectivity property of neural networks trained with stochastic gradient descent (SGD) 36 12 Model Soups [52] proposes to combine many independent runs with varied syper-parameter configurations. Similarly, DIWA [45 utilizes this idea of Model Soups while theoretically analyzing the importance of training different models with diverse hyper-parameter within mild ranges. Soups- bases methods [52] 43 rely on aggregating diverse models improve model generalizability. To induce greater diversity, some methods such as [34] using a high constant learning rate, (51) minimizing cosine similarity between weights, (20] using tempered posterior averages diverse model trained from auxiliary datasets.
not only efficient ways to aggregate
and model Ratatouille
42)
E Experiment Details
E.1 Experimental Setup Details
Dataset. We validate the effectiveness of our proposed method with four datasets, FMNIST 533
CIFAR-10 (26). Digit-5 [3],0] and DomainNet [40] The Pashion-MNIST (FMNIST) dataset is a dataset of Zalando's s article images consisting of training set of 60, 000 examples and test set of 10, 000 examples. Each example is a 28 x 28 grayscale image of a piece of clothing. The dataset is divided into 10 classes: t-shirt/top. trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, and ankle boot. The CIFAR-10 dataset is a popular dataset for machine learning research. It consists of 60, 0.000033 32 x 32 color images divided into 10 classes: airplane, automobile, bird, cat, deer, dog, frog. horse, ship, and truck. The dataset is split into 50, 000 training images and 10, 000 test images. The Digit-5 dataset is a collection of five popular digit datasets, MNIST [10] (55000 samples), MNIST-M (55000 samples), Synthetic Digits [3 (25000 samples), SVHN (73257 samples), and USPS (7438 samples). Each digit dataset includes a different style of 0-9 digit images. The DomainNet dataset is a large-scale dataset of images collected from six different domains: clipart, infograph, painting, quickdraw, real, and sketch. The dataset contains 600, 000 images, each labeled with one of 345 object categories. The images in the DomainNet dataset are of high quality and are diverse in terms of their content and style.
Model. We used the pre-trained models from the timm repo which are a collection of state-of-the art deep learning models for computer vision tasks. For our proposed LSS. we use Adam optimizer with a lcarning rate of 5e-4, momentum 0.9, and weight decay 5e-4. The default number of averaged models 4. Each model updates 8 epoch then aggregates with the others. The default affinity term coefficient is 3 and diversity term coefficient is 3. We set the batch size to 64 by default. For vision transformer (ViT) [1] model, we adopt ViT base model with 224 x 224 image size and 16 x 16 input patch size. The VIT is a neural network architecture for image classification that uses a self-attention mechanism to learn the relationships between pixels in an image. ViT has been shown to achieve state-of-the-ar results on a variety of image classification benchmarks, including ImageNet and C0.
Training Details. We implement all the methods in PyTorch, and we run all the experiments on an NVIDIA Tesla V100 GPU. Unless otherwise specified, the model performance in the experiments below refers to the global model performance after aggregation on the server side. For commonly used FL methods, due to the significant increase in local update steps that leads to worse convergence, we set their local update steps to 8.
Applying WAFT to FL Local Update. For SWA (21) SWAD M.. and our method LSS, we take more local update steps, with each model being averaged trained 8 steps, and the default number
19

---

we trained 32 models and each model trained 8 steps. The hyper-parameter configuration for model selection includes learning rate (1e-44,5 5e-5ee-- 4, le 5)), batch size ((32, 64, 128), dropout rate (0.0, 0.1, 0.3)). and weight decay [5e-4, 5e-5, 5e 56-66. 6]. Each run randomly select one of the hyper-pane parameee options. From each run of WAFT method, we take the weights of the epoch with maximum accuracy on the validation dataset, which follows the training distribution.
of models to be averaged is 4. For the Model Soups 52 method and DiWA 43
E.2 Extended Experiment Results
Arbitrarily increasing local steps cannot reduce communication rounds.
From T we see that simply increasing local steps does not always lead improved model performance. For FedAvg on the CIFARIO dataset, increasing local steps beyond actually results in decrease in model performance.
Table 4: FedAvg with different local steps: Label shift test accuracy after R communication rounds (CIFAR-10 with 5 Clients).
Method
Accuracy (r = 1t Accuracy (r = D Accuracy (r = 8) Accuracy (r = 12) Accuracy (r = 16)
FedAvg 35
31.03(2.84)
49.08(1.51)
58.34(0.86)
53.76(0.89)
53.21
Computational and memory costs comparison.
In Table 5 we provide detailed information on computational overhead and memory usage for various methods. Since the computational overhead and memory usage of FedAvg and other used FL methods are nearly identical, we only present the data for FedAvg here. Similarly, as the computational overhead and memory usage for SWA and SWAD, as well as for Soups and DiWA, are also nearly the same, we only show the data for SWA and Soups methods. It can be observed that our method requires more memory compared to other soups-based methods. However, the overall computational time for a single client's 's communication round is faster in our approach. This is because other soups-based methods require training large number of models repeatedly achieve good model performance. For instance, Soups needs to train 32 models, whereas our method only requires training 4 models. the number models trained by Soups is reduced to just 4, it only brings about a 5% improvement compared to FedAvg with communication round of 1.
Table 5: Computational and memory costs of different types of method (ResNet-18))
Costs
FedAvg [35]
SWA 21
Soups [52]
LSS (M = 2)
LSS (M = 4)
MACs (G)
1.82
1.82
1.82
2.73
4.55
Train Time Per Epoch (s)
2.66
2.73
2.66
12.27
20.43
Train Time Per Round
21.28
433.31
683.52
100.98
169.77
LSS encourages smoothness (reducing 3). In Table 6 we provide the performance degradation of trained models evaluating under varying levels of random noise. Generally, a smaller performance degradation indicates a more robust model, which to some extent reflects the smoothness of the trained model. We can observe that our method exhibits greater robustness noise perturbation.
Table 6: Smoothness of the trained model. Evaluated trained model performance drop on testset with added lp norm random noise. CIFAR-10 dataset Dirichlet distribution a 1.0 and a 0.1: Label shift test accuracy after R
CIFAR-10 (4/255)
CIFAR-10 (8/255)
Method
Accuracy (R = 1)
Accuracy (R 3) 4
Accuracy (R 1)
Accuracy (i = 3) 1
FedAvg 35
1.30
1.17
3.06
2.93
LSS
0.89
0.76
2.37
1.85
LSS improves flatness of loss landscape. The sharpness measure utilized in the Table 7 computes the median of the dominant Hessian eigenvalue across all training set batches through the Power Iteration algorithm [54]. This metrie signifies the maximum curvature of the loss landscape, commonly
20

---

employed in the literature on flat minima [23] to indicate sharpness. As demonstrated in the presented table, it is clear that our proposed method results in flatter minima compared to FedAvg.
Table 7: Loss landscape flatness quantification with Hessian eigenvalue.
FedAvg +
LSS (M = 2) + LSS (M 3)
LSS (M 4) 4
Hessian Eigenvalue
193.18
147.20
136.67
119.14
Evaluation with more clients. To assess the effectiveness of our method larger-scale client scenarios, conducted an expanded experiment involving 50 clients. From the Table we observe that our proposed method maintains significant advantage across different client scales, particularly when the number of communication rounds is small (R 1).
Table 8: Different client numbers (5 Clients and 50 Clients): Label shift test accuracy after R = and R communication rounds.
CIFAR-10 (5 Clients)
CIFAR-10 (50 Clients)
Method
Accuracy (R 1)
Accuracy (R f
Accuracy (R 1)
Accuracy = 3)
FedAvg
35
(8.34
66. 66.74(0.76)
49.32(0.93)
(8.39
LSS
65.56(1.50)
75.16(1.07)
56.72(0.53)
(0.46)
Table Different Network Architecture (ResNet-18 and ViT); Label shift test accuracy after and R communication rounds.
CIFAR-10 (ResNet-18)
CIFAR-10 (ViT Base)
Method
Accuracy (R 1)
Accuracy (R f
Accuracy (R 1)
Accuracy (R = 3)
FedAvg 35
(0.84().86)
66.74 66.74(0..6))
60.35(0.82)
69.38(0.51)
LSS
65.96(1.50)
75.16(1.07)
67.48(0..7)
76.81(0.477
Evaluation with VIT. To validate the effectiveness of our method across different network architectures, we conducted an expanded experiment using the Vision Transformer (ViT) model based on the Transformer architecture. Upon observing the T it evident that our method consistently enhances the commu- nication efficiency of federated learning with VIT model architectures.
FedAya Eveluation Compartson with Feddvg and LSS
God
3.08
Evaluation with different Non-lID level. To fur- ther comprehensively validate the effectiveness of our method under different levels of data hetero- geneity, we conducted experiments on the CIFAR- dataset by adjusting the coefficients of the Dirichlet distribution. We examined the performance of our method in scenarios with greater distribution varia- tions. Based on the Table 10 is evident that our method maintains significant advattgge in scenarios with larger data heterogeneity.
Mechadde
Figure FedAya Evaluation Comparison with FedAvg and LSS. Our method, LSS, when applied to large language models for instruction tuning, achieves higher This suggests that L.SS promising approach for improving performance and convergence in feder- ated learning settings for large language models, in addition its success image classification. Exploring the use of method a diverse set of complex LLM tasks is an interesting direction for future research.
than the common FedAvg.
Evaluation with different Initialized Models. To compare the performance of our method under differ- ent types of parameter initialization, we conducted experiments on the CIFAR-I0 10 dataset using both pre- trained and random initialization. Table []] shows that our method still maintains a significant advantage with random initialization, but does not achieve the near-optimal performance seen with pre-trained initialization.
21

---

Table 10: Different Non- IID level (Dirichlet distribution a 1.0 and a 0.1): Label shift test accuracy after R = 1 and R 3 communication rounds.
CIFAR-I0 (a 1.0)
CIEAR-IO (a = 0.1)
Method
Accuracy (R 1)
Accuracy 3)
Accuracy (R=
Accuracy (R = 3) t
FedAvg
35
(8.84
66.74(0.76)
18.30(2.25)
(1.85(1.2)
LSS
65.96(1.50)
75.16(1.07)
26.70(1.62)
(0.02
Comparison of Convergence Speed Between FedProx and LSS. Fig. Fig.Dsii shows the accuracy of the testing during the early and late phases in terms of the number of communication rounds required to reach convergence. These results demonstrate that our method outperforms FedProx in both the early and late phases of federated learning.
Evaluation of Large Language Models for Multilingual Instruction Tuning
Setup. We follow the setup of Fed-Aya 55 which involves four iterative steps: server-to-clien model downloading, local model training, client-to-server model uploading, and global model aggregation. For instruction tuning, we use the parameter-efficient fine-tuning technique, LoRA [9], applied to the Llama-7b model.
Dataset. We use the Aya dataset a multilingual instruction tuning dataset with annotations from contributors worldwide. Our experiments include 6 high-resource languages (English, Spanish, French, Russian, Portuguese, Chinese) and low-resource languages (standard Arabic, Telugu). The dataset is filtered to include contributors with at least 100 annotations, resulting in 38 clients with total of 25k data samples.
Model. The model used for our experiments is the Llama2-7b 48], fine-tuned using the LoRA technique. We evaluate the effectiveness of the training methods using an in-domain evaluation metric termed Ref-GPT4 [60], where GPT-4o rates the generated responses against ground truth responses. The score given by GPT-Ref ranges from 0 to 10. We adopt the same prompt template used in Fedl. ..M-Bnchh (55] The implementation of applying our method to LoRA is the same as that used in the ViT experiments (see Fig. 4 described earlier.
Result. Our method, LSS, when applied to large language models for instruction tuning, achieves higher scores than the common FedAvg. This suggests that LSS a promising approach for improving performance and convergence in federated learning settings for large language models, in addition to its success in image classification. Exploring the use of our method in a diverse set of complex LLM tasks is an interesting direction for future research.
Table 11: Different model initialization (Pre-trained v.s. Random): Label shift test accuracy after R = 1 and R = 3 communication rounds. Result: It shows that our method still maintains a significant advantage with random initialization, but it does not achieve the near-optimal performance seen with pre-trained initialization.
CIFAR-10 (Pre-trained)
CIFAR-10 (Random)
Method
Accuracy (R 1) Accuracy (R = 3)
Accuracy (R = 1)f
Accuracy (R = 3) t
FedAve 35
58.34(0.88)
66.74 66.74(0..6)
14.83(2.03)
25.42 (0.72(
LSS
65.56(1.50)
75.16(1.07)
30.64
37.86 (1.33)
CITW--10 LLata Phene!
Rigit5 Late Phans)
1.3
(b)
(c)
(d)
Figure 9: Convergence comparison of our proposed LSS with FedProx. LSS also achieves high accuracy much earlier (around 6 to 8 rounds) than FedProx, which takes hundreds of communication rounds.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The concepts of federated learning settings are clearly introduced and defined in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper acknowledges several limitations and areas for future research in Sec. 5) Firstly, it highlights trade-off between training memory and performance when reducing communication rounds in model merging. This indicates an awareness of the potential drawbacks of their method and suggests that further work is needed to make the method more training- maining-memory efficient. Additionally, the paper notes that it focuses exclusively on vision-related tasks, suggesting that extending the findings to language tasks or multimodal scenarios would be promising direction for future research.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
23

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, the paper meticulously outlines all relevant assumptions and presents com- prehensive, logically sound proofs, ensuring the validity and reliability of each theoretical result. These elements are included in Sec. and Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses all the information needed to reproduce the main experimental results. This information is provided in Sec. with additional detailed information available the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, the paper specifies all the training and test details necessary understand the results, including data splits, hyperparameters, their selection process, and the type of optimizer used. This comprehensive detailing ensures that the experimental setup and outcomes are transparent and reproducible in Section 4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ciiipss.cc/pulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary understand the results, including data splits, hyperparameters, their selection process, and the type of optimizer used. These details are comprehensively documented in Sec. 4ooo ensure clarity and reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
25

---

7. Experiment Statistical Significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the paper reports bars and other appropriate information about the statistical significance the experiments in Sec. These details are suitably and correctly defined ensure the reliability and validity of the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Yes, the paper provides sufficient information on computer resources needed to reproduce the experiments in Sec. 4 This includes details on the type of compute workers, memory requirements, and execution time, ensuring that others can accurately replicate the experimental setup and results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics / http///rrp...pppppp//ttttttttttt
Answer: [Yes]
26

---

Justification: Yes, the research conducted in the paper conforms, in every respect, with the NeurIPS Code Ethics. The paper adheres to the guidelines on responsible release and publication strategy, ensuring that all necessary safeguards are in place for controlled use of the model (see Sec.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: es, the paper discusses both potential positive and negative societal impacts of the work performed. The discussion includes an analysis of how the research can benefit society and also addresses possible adverse effects, ensuring balanced and comprehensive evaluation of the societal implications in Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not describe safeguards for the responsible release of data or models that have a high risk for misuse, such as pretrained language models, image generators, or scraped datasets. Because we does not focus the release of such assets, we focus model training techniques. (see Sec. 4.
Guidelines:
The answer NA means that the paper poses such risks.
27

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
The paper does not utilize external assets such as code, data, or models from other creators original owners. Therefore, there are no specific credits, licenses, or terms of use that need to be mentioned or respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi paperswithcode thcode. com /atasets
has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new assets such as code, data, or models. Therefore, there is no documentation provided alongside new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
28

---

Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments or research with human subjects. Therefore, it does not include instructions given participants, screenshots, or details about compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
The paper does not involve any study participants, crowdsourcing experiments, research with human subjects. Therefore, it does not describe potential risks incurred by study participants, disclose such risks to subjects, or obtain Institutional Review Board (IRB) approvals.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "6hHQfR0H4E", "reviewer": "Reviewer_2F5f", "summary": "This paper proposes LSS, a model interpolation-based local training technique to reduce the number of communication rounds required. The intuition is to regularize local models to connected low-loss valleys, so the aggregated model may have lower loss. LSS is empirically evaluated on a variety of datasets and types of distribution shifts.", "questions": "- Although Figure 2 is intuitive, it might not be so rigorous: high affinity and high diversity do not necessarily guarantee an aggregated model with low loss. In the example, two client’s local models are different “horizontally”, while the major axis of the loss landscape is also horizontal. However, this is not always guaranteed. If the clients models are different “vertically”, the proposed method may just fail. Mathematically, in algorithm 1, the gradient of $dist(f_{p_i}, f_p)$ and $dist(f_{p_i}, \\mathcal{M})$ is very likely to be nearly orthogonal. Could you explain intuitively why LSS is expected to perform well in most of the cases? \n- The scope of this paper is limited to FL with a pre-trained model. I understand that the pre-trained model may be larger in average, which makes convergence more changing. I am curious how LSS is limited to pre-trained models and whether it can generalize to randomly initialized models.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Figure 1 and 2, though may not be very rigorous, are clear and provide intuition to the readers. \n- The proposed algorithm is tested on a variety of datasets and types of distribution shifts.", "weaknesses": "- Readability: the notation in section 3 is not clear enough. For example, $n$ is used for both number of data (section 3.1) and number of averaged model (Alg 1), which is confusing. \n- (minor) It might be a little abuse of notation when using $\\mathcal{D}_i$ for both distribution and dataset. I suggest using different notations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We appreciate that you found our figure illustrations clear and our experimental scope comprehensive, covering diverse datasets and types of distribution shifts. We also value your suggestions for improving our manuscript.\n\nWe have addressed your concerns as follows:\n\n**1. Notation Clarity**\n\n**Revision:** \n- We will revise Algorithm 1, changing the interpolated model number notation from \"n\" to \"N\" (Line 4 in Algorithm 1) to distinguish it from the sample number.\n- In Section 3, we will provide different notation and description for distribution and dataset to prevent any potential confusion. Change Line 136 to 137 into \n\"For each client, we have access to `$n`$ training data points in the form of `$(\\mathcal{X}_i, \\mathcal{Y}_i) = \\{(x_{j}^{i}, y_{j}^{i})\\}_{j=1}^{n}`$, where `$y_{j}^{i}`$ denotes the target label for input `$x_{j}^{i}`$. \"\n- We will add a reference note in Section 3.1 directing readers to the notation table in Appendix A for convenient access to corresponding notations and descriptions.\"\"\n\n**2. Question: Could you explain intuitively why LSS is expected to perform well in most of the cases?**\n\n**Response.**\nWe acknowledge that high affinity and high diversity do not necessarily guarantee an aggregated model with low loss due to the complexity of the high-dimensional loss landscape. However, our empirical evaluations show that using pre-trained model initialization, combined with our proposed random interpolation and the terms affinity and diversity, positively impacts performance and convergence across various types of distribution shifts.\n\n**Clarification.** \nOur method performs well in most cases when using overparameterized pre-trained models. Typically, training with a pre-trained model results in small initial gradient dissimilarity, and overparameterization ensures that the optimal parameters are close to the initialization. See Section 3 (see **Line 181 to 183**) for more detailed explanations. In summary, **small gradient dissimilarity among clients** and a **small parameter distance from initialization to optima**, combined with leveraging our introduced diversity and affinity terms, make our method work well in most cases when training with pre-trained models.\n\n\n**3. Limitation: The method is tested only with pre-trained models. It is unclear how it would perform with randomly initialized models.**\n\n**Response.** Our LSS approach is also effective in improving performance and convergence when training models with random initialization. However, it does not achieve the impressive few-round near-optimal performance seen when training with pre-trained models.\n\n**Added Experiment.** In response to your query, we have added performance results comparing the training with randomly initialized and pre-trained parameters in the **Author Rebuttal Attached File Table 11**.\n\nThank you again for your valuable feedback. We believe that these revisions significantly strengthen our manuscript, and we appreciate your feedback in making these improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jEylmdFSFb", "reviewer": "Reviewer_3kGS", "summary": "This paper proposes a method called Local Superior Soups (LSS), a novel technique for model merging in cross-silo federated learning aimed at reducing communication rounds while enhancing model performance. This paper introduces random interpolation, diversity term, and affinity term to alleviate the need for time-consuming model selection and redundant model training. Rigorous experiments on 4 datasets with 11baselines demonstrate the effectiveness of LSS.", "questions": "1. In Figure 3, how do the results of LSS compare with those of FedProx? Given the similarities in the core concepts between LSS and FedProx, such a comparison would be insightful for evaluating the distinct advantages of LSS.\n2. The results presented in Tables 1 and 2 suggest that LSS performs well in the initial training rounds. Can the authors clarify whether LSS is primarily advantageous only during these initial rounds? \n3. Furthermore, how should the training process be continued post-initialization? Is it feasible to employ LSS throughout the entirety of the training process, or would alternative methods be more effective in later stages?", "limitations": "Please see the weaknesses and the questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper discusses the importance of bridging two low-loss valley to reduce communication rounds.\n2. This paper introduces two quantifiable metrics, diversity and affinity, which serve as indicators of model quality..\n3. This paper conducts extensive experiments to illustrate the effectiveness of LSS.", "weaknesses": "1. The distinction between LSS and similar federated learning methods such as FedProx, which also incorporates weights from global models to regulate client loss, is not clearly discussed.\n2. The subsection 3.3.1 titled \"Random interpolation conserving connected low-loss region.\" lacks mathematical detail to fully understand the interpolation process.\n3. The requirement for clients to receive the interpolated model pool ($M$) could potentially lead to significant communication overheads, which may not present a clear advantage over simpler methods like FedAvg or FedProx.\n4. The connection between Theorem 3.1 and the core methodology of LSS, specifically the diversity and affinity terms, appears tenuous. These terms do not seem to be directly derived from the theorem, which may weaken the theoretical foundation of the proposed method.\n5. This paper should consider referencing relevant literatures or conducting preliminary experiments to support its statments on the part called \"Limitation of previous model soups methods\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback on our paper. Thank you for your positive comments on the comprehensiveness of our experiments and for recognizing the key contribution of our proposed two metrics (i.e., affinity and diversity) in quantifying model quality for model selection in model interpolation.\n\nBelow are our responses to the key points raised:\n\n**1. Comparison with Similar Methods**\n\n**Response.** \nWe have discussed the differences between our method and the most relevant existing methods (model averaging) in the Method and Experiment section (see **Line 202 to 220** and **Line 300 to 302**). Our proposed LSS consists of three novel key components: random model interpolation, diversity regularization, and affinity regularization, which is fundamentally different from FedProx.\n\n**Clarification.** \nCompared with FedProx, our LSS contains random model interpolation and diversity regularization two novel components that help reduce the communication cost, the only similarity between LSS and FedProx is the presence of a regularization term: ours leverages affinity regularization, while FedProx uses proximal regularization. However, they **differ significantly in their motivation, implementation, and focus**.\n    \n(a) **Motivation**: The affinity term in LSS is designed to ensure that the expansion of low-loss regions remains close to the shared initialization point when incorporating diversity regularization. This increases the likelihood of overlapping connected regions (see **Line 81 to 91**), thereby enhancing performance and convergence by leveraging pre-trained models and significantly reducing communication rounds. In contrast, FedProx aims to improve general convergence and performance in the presence of statistical heterogeneity.\n\n(b) **Implementation**: In LSS, the affinity term minimizes the distance of local models from the **initialized model parameters** (i.e., the pre-trained model parameters), **rather than the global model** (see **Line 81 to 83**, **Algorithm 1 Line 2** and Require Description). In contrast, FedProx directly incorporates global model weights to regulate client loss.\n\n(c) **Focus**: FedProx primarily introduces a regularization term to address general convergence and performance issues arising from statistical heterogeneity. In contrast, our method focuses on leveraging pre-trained models to achieve significant communication round savings. We approach this from a novel perspective, using model interpolation techniques to enhance FL convergence and performance. Additionally, we introduce an affinity regularization term to improve model quality during interpolation. Our **affinity term is intrinsically linked to the diversity term and model interpolation process** (see **Line 81 to 94**, and **Line 252 to 255**), making it an **integral and non-isolated part of our approach**.\n\n**Added Experiment.**\nTo clearly illustrate the distinct advantages of LSS, we have included comparative results with FedProx similar to those in Figure 3 in the **Author Rebuttal Attached File. Fig.8.** This comparison highlights the benefits of our approach in terms of communication efficiency and performance.\n\n**2. Mathematical Detail of Random Interpolation: The subsection 3.3.1 lacks mathematical detail to fully understand the interpolation process.**\n\n**Response:** We will revise subsection 3.3.1 to provide a more detailed mathematical explanation of the random interpolation process. \n\n**3. Communication Overhead: Potential significant communication overhead due to the requirement for clients to receive the interpolated model pool.**\n\n**Clarification:** We would like to clarify that our LSS method does not lead to additional communication overhead. Since we use the same model interpolation operation after model selection as model soups, **the local model is merged from the interpolated model pool before communication**. This ensures that our method maintains the same level of overhead as basic methods like FedAvg. In our revised version, we will introduce more background on model soups in Section 1 to improve clarity regarding communication overhead in parameters.\n\n**4. Theoretical Connection**\n\n**Response.** Theorem 3.1 aims to provide a convergence guarantee when our proposed regularization terms are added, not to derive these terms directly from the theorem. In revision, we will clarify the motivation and role of the regularization terms at the beginning of Theorem 3.1. \n\n**5. Reference and Statement Support**\n\n**Response.** We would like to clarify that we have already included references to support our statements in Section 3 (see **Line 203**). Although the limitation is not explicitly stated in the references, it can be clearly derived from the algorithm process. In revision, we will expand this part by adding more related work to better support our claims.\n\n\n**6. Experiment Scope: The results presented suggest LSS performs well in initial training rounds. It is unclear if LSS is advantageous throughout the training process.**\n\n**Response:** Yes, as indicated in Figure 3, our LSS performs well not only during the initial training rounds but also throughout the entire training process. Additionally, we would like to highlight that LSS is designed to reduce communication rounds when using pre-trained models. Training with 'continued post-initialization' is not the focus of our problem setting.\nSpecifically, we aim to emphasize that LSS is tailored to minimize communication rounds in the context of pre-trained models. However, the design of our method should be compatible with existing algorithms for post-initialization training. Examining the effects of post-initialization training falls beyond the scope of our current paper, we will explore this topic in our future research work.\n\n\nWe believe these revisions significantly enhance the clarity of our paper. Thank you again for your insightful comments and suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zpDrB6RD5V", "reviewer": "Reviewer_9KZz", "summary": "In this paper Local Superior Soups (LSS) is proposed to minimize communication rounds in federated learning (FL) using pre-trained models, specifically tackling data heterogeneity challenges. LSS achieves this by employing sequential model interpolation, maintaining connectivity, and integrating diversity and affinity regularization terms. These innovations enable more local training steps and fewer communication rounds, effectively preventing client drift. Designed for adapting pre-trained models in FL, LSS enhances training efficiency, making it well-suited for deployment in edge computing applications.", "questions": "None.", "limitations": "See weakness part", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method can effectively reduce the communication rounds in federated learning (FL) using pre-trained models.\n\n2. The proposed method seems sound.\n\n3. This paper is well written.", "weaknesses": "1. Only two small-scale image datasets are used in experiments. More large-scale datasets, especially those in other modalities, should be used.\n\n2. More pretrained models should be explored.\n\n3. More tasks besides image classification should be incorporated into experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback on our manuscript. We appreciate your positive comments on the effectiveness and soundness of our proposed method, as well as the clarity of our writing.\n\n**Expermental Scope Clarification.**\nWe would like to clarify that the current set of experiments is sufficient to support our claims and demonstrate the effectiveness and generalizability of our proposed method. Additionally, other reviewers have acknowledged the comprehensiveness of our experiments. Reviewer 2F5f noted, **\"The proposed algorithm is tested on a variety of datasets and types of distribution shifts,\"** and Reviewer 3kGS commented, **\"This paper conducts extensive experiments to illustrate the effectiveness of LSS.\"**\n\nHowever, we also recognize that extending our experiments could further enhance our experimental scope, particularly with initial explorations on large language models (LLMs).\n\n We have addressed your concerns as follows:\n\n**1. Issue: Additional Evaluations on Large-Scale Models besides Image Classification**\n\n**Action Taken.**\nWe evaluated our proposed method, LLSS, against FedAvg on a large language model for a multilingual instruction tuning task to demonstrate the generalizability of LSS. Additional details can be found in the Appendix.\n\n**Experiment Detail.**\nEvaluation of Large Language Models for Multilingual Instruction Tuning.\n**Setup.** We follow the setup of Fed-Aya [1], which involves four iterative steps as common federated learning: server-to-client model downloading, local model training, client-to-server model uploading, and global model aggregation. For instruction tuning, we use the parameter-efficient fine-tuning technique, LoRA applied to the Llama2-7b model.\n\n**Dataset.**\nWe use the Aya dataset [1], a multilingual instruction tuning dataset with annotations from contributors worldwide. Our experiments include 6 high-resource languages (English, Spanish, French, Russian, Portuguese, Chinese) and 2 low-resource languages (standard Arabic, Telugu). The dataset is filtered to include contributors with at least 100 annotations, resulting in 38 clients with a total of 25k data samples.\n\n**Model.**\nThe model used for our experiments is the Llama2-7b, fine-tuned using the LoRA technique. We evaluate the effectiveness of the training methods using an in-domain evaluation metric termed Ref-GPT4, where GPT-4o rates the generated responses against ground-truth responses. The score given by GPT-Ref ranges from 0 to 10. We adopt the same prompt template used in FedLLM-Bench [1]. The implementation of applying our method to LoRA is the same as that used in the ViT experiments (see Fig.4 described earlier).\n\n**Result.**\n**In Figure 9 in the Author Rebuttal Attached File**, our method, LSS, when applied to large language models for instruction tuning, achieves higher scores than the common FedAvg. This suggests that LSS is a promising approach for improving performance and convergence in federated learning settings for large language models, in addition to its success in image classification. Exploring the use of our method in a diverse set of complex LLM tasks is an interesting direction for future research.\n\n[1] Ye, Rui, et al. \"FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models.\" arXiv preprint arXiv:2406.04845 (2024).\n\n\n**2.  Issue: Additional Evaluations on More Pre-trained Models**\n\n**Clarification**\nWe have added evaluations on various pre-trained models, including ResNet18, ResNet50, ViT, and LLMs (llama2-7b). We believe our experimental evaluation covers a diverse range of model architectures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the AC and reviewers for their valuable time. We are delighted to see that the reviewers highlight our **"paper is well written"**, recognizing the importance of our idea of **"bridging two low-loss valley to reduce communication rounds"**, and our proposed method **"effectively reduce the communication rounds"** and **"tested on a variety of datasets and types of distribution shifts."** 

This rebuttal clarifies 2F5f’s concern on the notation usage, 3kGS’s questions regarding our method design and motivation, 9KZz’s concerns about validation on more datasets, and addresses all other minor questions and discussion points from all reviews.

Below is a summary of our rebuttal:

**Response to Reviewer 2F5f:**

1. **Notation Clarity:**
Revised notation in Algorithm 1 and Section 3 for better clarity. Added a reference note to the notation table in Appendix A.

2. **Explanation of LSS Performance:**
Clarified that training with pre-trained models and regularized model interpolation promotes a low-loss connection, enhancing performance. 

3. **Limitation of Pre-trained Models:**
Clarified that LSS is effective with random initialization but excels with pre-trained models. Added comparative results in the attached file.

---

**Response to Reviewer 3kGS:**

1. **Comparison with Similar Methods:**
Differentiated LSS from FedProx in terms of implementation, motivation, and focus. Added comparative results in attached file highlighting LSS's advantages.

2. **Mathematical Detail of Random Interpolation:**
Revised subsection 3.3.1, including detailed mathematical explanations and relevant formulas. 

3. **Communication Overhead:**
Clarified that LSS maintains similar communication overhead to methods like FedAvg.

4. **Theoretical Connection:**
Clarified the role of diversity and affinity terms in supporting convergence guarantees in Theorem 3.1. 

5. **Reference and Statement Support:**
Included additional references to support our claims.

6. **Experiment Scope:**
Clarified LSS's effectiveness throughout the training process.

---

**Response to Reviewer 9KZz:**

1. **Experimental Scope Clarification:**
Added evaluations on a large language model for multilingual instruction tuning. Clarified that various pre-trained models (ResNet18, ResNet50, ViT, LLM) are included to demonstrate the generalizability of LSS.

Meta-review: All three reviewers have provided positive feedback on this submission. The paper introduces a novel method to reduce communication costs in federated learning for pre-trained models. The authors have adequately addressed the concerns and weaknesses raised by the reviewers. Based on this, I recommend acceptance.
Decision: Accept (poster)