One-Shot Sequential Federated Learning for Non-IID Data by Enhancing Local Model Diversity

Authors: naibo  wang
yuchen  deng
wenjie  feng
shichen  fan
jianwei  yin
see-kiong  ng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 13:14:23

Abstract: Traditional federated learning mainly focuses on parallel settings (PFL), which can suffer significant communication and computation costs. In contrast, one-shot and sequential federated learning (SFL) have emerged as innovative paradigms to alleviate these costs. However, the issue of non-IID (independent and identically distributed) data persists as a significant challenge in one-shot and SFL settings, exacerbated by the restricted communication between clients. In this paper, we improve the one-shot sequential federated learning for non-IID data by proposing a local model diversity-enhancing strategy. Specifically, to leverage the potential of local model diversity for improving model performance, we introduce a local model pool for each client that comprises diverse models generated during local training, and propose two distance measurements to further enhance the model diversity and mitigate the effect of non-IID data. Consequently, our proposed framework can improve the global model performance while maintaining low communication costs. Extensive experiments demonstrate that our method exhibits superior performance to existing one-shot PFL methods and achieves better accuracy compared with state-of-the-art one-shot SFL methods on both label-skew and domain-shift tasks (e.g., 6\%+ accuracy improvement on the CIFAR-10 dataset).

Main paper: 
 introduction: 
 Federated learning (FL) [40] is a promising paradigm which enables collaborative machine learning [2, 50] amongst multiple clients to build a consensus global model without the need to access others' datasets. This paradigm offers salient benefits such as preservation of privacy [17] , security of data [39] , and the facility for different clients to derive a model exhibiting a higher degree of inference capability compared to individual client-based training [55] .As shown in Fig. 1 , two federated learning paradigms are parallel FL (PFL) [5, 41] and sequential FL (SFL) [38] . PFL synchronizes model training across clients in parallel, such as FedAvg [40] , while SFL adopts a client-by-client training sequence that is widely applied in many scenarios such as healthcare [8, 9] . Compared with PFL, SFL exhibits significant advantages in training efficiency, as evidenced by a reduction in training rounds [57] . SFL also shows robust performance with limited datasets [28] and offers enhanced data privacy protection due to its decentralized architecture [21] .One main concern of SFL is communication costs. To mitigate communication costs, one-shot federated learning [19] has been proposed where only one communication round is needed for the clients to interact with the server or other clients. However, current one-shot FL works [22, 58] mainly focus on PFL (Fig. 1 (a) ) which typically requires a central server to facilitate model training. This can lead to numerous limitations including risk of privacy leakage [27] and server node bottlenecks [20] . To address these issues, the decentralized structure [47] has been incorporated into one-shot PFL which typically utilizes a mesh-topology network for clients to disseminate their models to others. Nonetheless, this setting still leads to significant communication overhead than SFL. In contrast, one-shot SFL, wherein each client only needs to communicate with its adjacent client once, can greatly reduce communication costs. However, existing studies on one-shot SFL [8, 44] struggle with handling the non-IID data, a common challenge in FL that significantly impairs the performance of federated models [46] . How to tackle the non-IID data in one-shot SFL is still an open problem.The key question in one-shot SFL is how to effectively transfer knowledge given the limited communication, especially with the non-IID data. Our insight is to utilize the diversity of models. Existing studies have shown that combining multiple neural networks can notably enhance the model's performance due to the inherent diversity amongst model weights [7, 25, 52] . This observation has also been solidly backed by theoretical support from Rame et al. [42] . However, directly applying such a diversity strategy in SFL is not feasible, since SFL restricts each client to receive only one model from its adjacent client . This limitation could lead to insufficient diversity during model training, which can potentially undermine the performance of the final trained model. To overcome this challenge within the one-shot SFL framework, we propose a diversity-enhanced mechanism for model training. This mechanism is designed to augment the diversity of locally trained models. By generating a broader spectrum of models, we facilitate enriched knowledge transfer between adjacent clients, which also serves to mitigate the impact of data distribution disparities across clients.In this paper, we present a novel one-shot sequential Federated learning framework by Enhancing Local Model diversitY (FedELMY). Specifically, we improve the local model training diversity by constructing a model pool consisting of various models for each client. We introduce two distance regularization terms during the local training process to enhance model training diversity while mitigating the impact of non-IID data. Fig. 2 presents a case study illustrating our core concept. Compared with conventional federated learning methods, our approach effectively minimizes communication costs, safeguards data privacy, and mitigates the detrimental effects of non-IID data, thereby enhancing overall model performance. Experiments show that our method can outperform existing one-shot SFL methods on both label-skew and domain-shift datasets.The main contributions of this paper are as follows:• We tackle the novel and practical problem of one-shot sequential federated learning. To the best of our knowledge, this is the first work to systematically investigate the oneshot communication setting in sequential federated learning. One well-known PFL method is FedAvg [40] , whose performance is hindered due to the dispersed nature of the data (non-IID). Methods such as FedProx [36] , FedDyn [1] , Astraea [13] , pFedMe [49] , and FedCurv [46] use a measure of global parameter stiffness to tackle data heterogeneity in federated learning. Additionally, approaches like FedDC [16] and SCAFFOLD [29] use a global gradient adjustment mechanism to manage local variations of the data. Other methods like FedGMM [53] , FCCL [23] , FedBN [37] and ADCOL [34] utilize the personalized model instead of a shared global model to improve the performance of federated learning. These methods all necessitate a central server for model training.Another PFL scenario is the decentralized federated learning [3, 43, 56] when edge devices conduct training without a central server. Sun et al. [48] proposed the decentralized FedAvg with momentum (DFedAvgM) algorithm to improve the performance of trained models in the decentralized federated learning setting. Shi et al. [45] applied the SAM [15] optimizer to improve the model consistency of decentralized federated learning. This setting can better protect privacy than centralized PFL but incurs more communication costs. 
sequential federated learning: 
 Recently, Sequential Federated Learning (SFL) started to gain attention in the FL community. Micah J et al. [44] proposed a basic SFL framework to facilitate multi-institutional collaborations without sharing patient data; Li et al. [38] proved that the convergence guarantees of SFL on heterogeneous data are better than PFL for both full and partial client participation, and validated that SFL outperforms PFL on extremely heterogeneous data in cross-device settings; Chen et al. [9] proposed MetaFed, an SFL scheme with cyclic knowledge distillation for personalized healthcare; Cho et al. [10] also provided convergence analysis of sequential federated averaging and proves that it can achieve a faster asymptotic convergence rate than vanilla FedAvg with uniform client participation under suitable conditions. However, existing works still lack the ability to deal with non-IID data and exhibit poor performance in real-world applications.One-shot federated learning [12, 14, 19, 47] , which aims to train a global model with only one round of communication between the clients and the server, has been proposed to reduce the communication cost and simultaneously enhance data privacy protection in federated learning. FedDISC [54] made progress in one-shot semi-supervised federated learning, employing a pre-trained diffusion model. Investigations by Diao et al. [11] and Joshi et al. [26] have independently explored one-shot federated learning, viewing the problem through the respective perspectives of the open-set problem and Fisher information. Moreover, the proliferation of pre-trained models has sparked interest in collaborative, modelcentric machine learning [4, 51] . Instances such as FedKT [33] and DENSE [58] are two one-shot federated learning schemes that choose to send models instead of gradients to the central server and generate a final global model through knowledge distillation. Despite their advancements, these one-shot methods are not designed for sequential federated learning, which still results in a higher risk of privacy leakage. Thus, devising a one-shot sequential federated learning framework is still under-explored. 
method: 
 In this section, we introduce our problem setting and present our proposed algorithm FedELMY. The overview of our method is demonstrated in Fig. 3 .Assume there are 𝑁 different clients (parties), each client has its own private dataset 𝐷 𝑖 = {(𝑥 𝑘 , 𝑦 𝑘 )} 𝑛 𝑖 𝑘=1 with size 𝑛 𝑖 . The principal goal of sequential federated learning is to develop a global model 𝑚 across the dataset D = {𝐷 𝑖 } 𝑁 𝑖=1 , by minimizing the error on training data. The optimization objective is formulated as:EQUATIONwhere 𝐿(𝑚; 𝑥, 𝑦) is the loss function evaluated on the private dataset 𝐷 𝑖 from client 𝑖 (𝑐 𝑖 ) with model 𝑚. The training procedure of sequential federated learning is described as follows:1. At the start of each training round 𝑟 , the client's training sequence{𝑐 𝜋 1 , 𝑐 𝜋 2 , • • • , 𝑐 𝜋 𝑁 } is determined by randomly selecting in- dices {𝜋 1 , 𝜋 2 , • • • , 𝜋 𝑁 } without replacement from set {1, 2, • • • , 𝑁 }.2. At the very beginning of training when round 𝑟 = 1, randomly initialize global model 𝑚 (0) .3. For the 𝑖-th client (𝑐 𝜋 𝑖 ) in round 𝑟 , initialize its model 𝑚(𝑟 ) 𝜋 𝑖 ,0with the latest model:EQUATIONwhere 𝑚 (𝑟 -1) is the global model received from round 𝑟 -1, and𝑚 (𝑟 ) 𝜋 𝑖 -1 ,𝐸 𝑙𝑜𝑐𝑎𝑙is the model received from the (𝑖 -1)-th client (𝑐 𝜋 𝑖 -1 ) in round 𝑟 after it trained its model for 𝐸 𝑙𝑜𝑐𝑎𝑙 epochs.The authors of SWAD [7] demonstrate that solely minimizing empirical loss in a single model is typically insufficient for achieving 𝑎𝑣𝑔 which is efficiently derived from the model pool M 𝑖 -1 of the previous client 𝑖 -1:EQUATIONThe parameters to be sent to the next client have been improved to an average of a local diverse model pool instead of a single model.Then, for each individual client 𝑖, we train an additional series of 𝑆 models, denoted as {𝑚 𝑖 𝑗 } 𝑆 𝑗=1 , on the local dataset 𝐷 𝑖 to probe a broader range of diverse models. Each new model, 𝑚 𝑖 𝑗 , stems from the existing models in the established model pool, M 𝑖 . More specifically, initialization of every new model 𝑚 𝑖 𝑗 is achieved by averaging all the models currently in circulation within the model pool M 𝑖 , with the first model 𝑚 𝑖 0 included; that is,EQUATIONIn this way, we craft a unique launching point for the training of each new model, ensuring a departure from all earlier models within our pool M 𝑖 . Consequently, a diverse array of starting points is harnessed, granting us access to a more comprehensive set of potential solutions that evade the restrictions of a singular training trajectory and its attendant limitations.After initialization, the model 𝑚 𝑖 𝑗 is then trained for 𝐸 𝑙𝑜𝑐𝑎𝑙 epochs, employing the same hyperparameters as with preceding models. Upon finishing training, 𝑚 𝑖 𝑗 will be added to the model pool M 𝑖 . After training all 𝑆 models for client 𝑖, the client will calculate the final averaged model 𝑚 𝑖 𝑎𝑣𝑔 , which is the average of all models in M 𝑖 , and send 𝑚 𝑖 𝑎𝑣𝑔 to the subsequent client, 𝑖 + 1. This process will be repeated until client 𝑖 = 𝑁 concludes its local training. Upon completion, the final client 𝑖 = 𝑁 will output the final model 𝑚 𝑓 𝑖𝑛𝑎𝑙 , which is the average of all models in its model pool M 𝑁 , i.e., 𝑚 𝑓 𝑖𝑛𝑎𝑙 ← 1| M 𝑁 | | M 𝑁 | -1 𝑡 =0𝑚 𝑁 𝑡 . To further enhance training diversity and prevent the convergence of the newly generated model 𝑚 𝑖 𝑗 towards the states of existing models in M 𝑖 -thereby diminishing diversity-we aim to maintain the current exploration direction of the model as distinct as possible from those of existing models, throughout the training process. Consequently, we introduce a new distance control term, 𝑑 1 , in the training process, which is defined as:EQUATIONwhere 𝑚 𝑖 𝑗 represents the current model being trained for client 𝑖, and 𝑚 𝑖 𝑡 signifies a model in the model pool M 𝑖 . We use the ℓ 2 norm for 𝑑𝑖𝑠𝑡, that is, calculating the averaging the 𝐿 2 -𝑛𝑜𝑟𝑚 between the current model and all existing models in the model pool. The loss function incorporates and deducts this distance during training, allowing the model to maximize its distance from the existing models, therefore promoting training diversity.In non-IID scenarios, variations in local objectives may risk deviating from the globally optimal solution across multiple local iterations, thereby impeding convergence [36] . To prevent significant deviation of the model from the global solution, we further L (𝑚 𝑖 𝑗 ) = ℓ (𝑚 𝑖 𝑗 ; 𝐷 𝑖 ) -𝛼 • 𝑑 1 + 𝛽 • 𝑑 2 12 𝑚 𝑖 𝑗 ← 𝑚 𝑖 𝑗 -𝜂∇ 𝑚 L (𝑚 𝑖 𝑗 ) 13 end 14 M 𝑖 ← M 𝑖 ∪ {𝑚 𝑖 𝑗 } 15 end 16 𝑚 𝑖 𝑎𝑣𝑔 ← 1 | M 𝑖 | | M 𝑖 | -1 𝑡 =0 𝑚 𝑖EQUATIONwhere 𝑚 𝑖 0 represents the first model in the model pool M 𝑖 for client 𝑖. The loss function also incorporates this term to ensure that the model maintains reasonable proximity to the initial model 𝑚 𝑖 0 (global solution of previous clients) in the pool during local updates. It helps mitigate the effect of non-IID data distribution while accommodating system heterogeneity.The total loss function L for model 𝑚 𝑖 𝑗 is formed as follows:EQUATIONwhere ℓ denotes the original loss function, 𝐷 𝑖 is the local dataset for client 𝑖, and 𝛼 and 𝛽 are two hyper-parameters that govern the effect of both distances on model training. Our aim is to achieve a balance that enhances model training diversity while mitigating the impact of non-IID data.The details of our method are depicted in Algorithm 1. The training procedure begins at client 1, which acts as the starting point for training with a warm-up phase (line 1). Then, each client 𝑖 will Communication cost. In our method, each client will send a model to its adjacent client only once, thus only 𝑁 -1 model exchanges are required. Therefore, the overall communication cost for all 𝑁 parties is 𝑂 (𝑁 𝑀), where 𝑀 represents the size of the model. Computation cost. In our approach, every client will train 𝑆 models, thus the overall computation cost for all 𝑁 parties is 𝑂 (𝑁 𝑆𝐸 𝑙𝑜𝑐𝑎𝑙 ). 
update model 𝑚 (𝑟 ): 
 𝜋 𝑖 ,0 for 𝐸 𝑙𝑜𝑐𝑎𝑙 epochs based on 𝐷 𝜋 𝑖 and send [6] is chosen as the optimizer, then the update can be described by:𝑚 (𝑟 ) 𝜋 𝑖 ,𝐸 𝑙𝑜𝑐𝑎𝑙 to the (𝑖 + 1)-th client (𝑐 𝜋 𝑖+1 ), if SGDEQUATIONwhere 𝑚 5. At the end of round 𝑟 , i.e., after all clients finished their training, we can get the global model 𝑚 (𝑟 ) of round 𝑟 as:EQUATIONFor one-shot sequential federated learning, only 1 training round is required, i.e., 𝑟 ≡ 1, thus 𝑚 (1) will be the final global model.Analysis. Although the communication cost is low, the information exchanged in SFL is limited, especially under the one-shot setting. Therefore, it is a big challenge to aggregate diverse and useful information to get a better global model. In other words, the diversity of distributed data under the one-shot SFL paradigm has not been well utilized. Meanwhile, the non-IID data in SFL increases the risk of models getting stuck in a local minimum, which further limits the enhancement of the generalization ability of the global model [36] . Hence, it is essential to explore methods to increase the diversity during model training and mitigate the impact of non-IID data to enhance the model's performance in one-shot SFL.Datasets and Data Partition. As shown in Fig. 4 , in federated learning applications, non-IID data distribution is a common scenario, often arising in two forms: label-skew [11] and feature-skew (also known as domain-shift) [24] . We seek to corroborate the effectiveness of our method under these two distinctive non-IID conditions. We select two datasets associated with label-skew setting, namely CIFAR-10 [30] and Tiny-ImageNet [31] , along with two datasets linked to domain-shift setting, specifically PACS [32] and Office-Caltech-10 [18] .The training datasets for CIFAR-10 and Tiny-ImageNet are partitioned into 𝑁 = 10 clients with Dirichlet distribution 𝐷𝑖𝑟 (0.5). Meanwhile, with respect to the PACS and Office-Caltech-10 datasets, they intrinsically contain four distinct domain samples, each allocated to a single client, yielding 𝑁 = 4 clients in total. Each client is provided with a randomly selected 90% of the local training dataset for training purposes, with the remaining serving as the validation set. We test the final model 𝑚 𝑓 𝑖𝑛𝑎𝑙 on all test data from all clients, providing a global test performance measurement. Baselines. We compare our method baselines under federated learning settings: DFedAvgM [48] , a decentralized adaptation of the extensively studied FedAvg [40] method in federated , a personalized sequential federated learning approach used for healthcare, which requires at least two rounds of communication; FedSeq [38] , which is the state-of-the-art sequential federated learning algorithm.In an effort to ensure a fair comparison, we adapted the decentralized PFL methods DFedAvgM and DFedSAM to the one-shot setting, and adjusted these methods to select all clients for training and communication to fit the setting. In the remaining parts, unless otherwise specified, we set the local training epoch 𝐸 𝑙𝑜𝑐𝑎𝑙 to 200. For more implementation details about our method and all baseline methods, such as the optimizer, learning rate and batch size we employed, please refer to Sec. 1 of the supplementary material.4.2.1 One-shot Setting. Table 1 presents the test accuracy of different methods in two training settings (𝐸 𝑙𝑜𝑐𝑎𝑙 = 100 and 200) to verify the robustness of our method. Results show that our FedELMY method outperforms all other methods across all datasets including both label-skew and domain-shift tasks. Specifically, FedELMY surpasses existing SFL methods by over 6% in accuracy, and outperforms current PFL methods by more than 15% on the CIFAR-10 (label-skew) dataset. Additionally, FedELMY achieves a 25% higher accuracy over PFL methods and at least an 1.5% improvement over SFL methods on the PACS (domain-shift) dataset. These results consistently demonstrate the effectiveness of our method.It is noteworthy that for the Tiny-ImageNet dataset, due to its large number of classes (200), most PFL baselines fail to deliver effective performance such as DENSE (close to random guesses), suggesting that they cannot effectively handle datasets with too many classes. However, our method works well and also achieves improvement compared to other SFL methods. Such improvements are attributed to the diverse local training so that our method can learn better feature representations.In addition, it is not always the case that the test accuracy of 𝐸 𝑙𝑜𝑐𝑎𝑙 = 200 is better than 100, indicating that simply increasing 2 , even under few-shot settings, FedELMY consistently outperforms the SFL baselines. Such observations underscore the effectiveness and scalability of FedELMY. Meanwhile, we observe that as the number of training rounds increases to a certain extent, the overall performance does not significantly improve, indicating that the model has reached a state of convergence. This concludes that blindly increasing the number of training rounds will not significantly enhance the model performance.In this section, we will evaluate our method's efficiency by comparing its communication cost and model performance under different computation costs with other baselines. Other methods mandating communication with their neighbors like DFedAvgM and DFedSAM, will require even higher communication costs. Accordingly, our method effectively diminishes the communication burden of FL and thus strengthens data privacy.Given that FedELMY requires training 𝑆 models at each client, it involves 𝑆 × 𝐸 𝑙𝑜𝑐𝑎𝑙 epochs of training per client. Compared to FedSeq, in which each client trains a single model for 𝐸 𝑙𝑜𝑐𝑎𝑙 epochs, and MetaFed, which necessitates 2 × 𝐸 𝑙𝑜𝑐𝑎𝑙 epochs of training per client, our method appears to demand a higher computational expense. However, FedELMY significantly enhances the model performance without increasing the communication cost (Table . 1). Thus, our method strikes a balance (trade-off) between computational cost (training time) and performance gains.Meanwhile, we conducted experiments to align the computational costs of different methods for a fair comparison (Fig. 6 ). First, we align the computation cost of our method to the baselines. By setting {𝑆 = 5, 𝐸 𝑙𝑜𝑐𝑎𝑙 = 40}, FedELMY maintains the same computational cost as baselines. As we can see, even under this setting, our method still outperforms the baselines; then, we align the computation cost of the baselines to FedELMY by increasing 𝐸 𝑙𝑜𝑐𝑎𝑙 to 1000, Fig. 6 shows that FedSeq displays worse performance and MetaFed shows negligible variation in its performance. This can be attributed to the phenomenon of overfitting due to excessive local training, which increases the model's generalization error and hence, compromises their performance. Contrastingly, our approach, which In this section, we examine the impact of distance metrics, hyperparameters, and diversity control measures on our method's performance, as well as the influence of client orders.We conducted ablation experiments to verify the effectiveness of the components in our method. Since the model pool M is indispensable, we examined the impact of the two distance regularization terms, 𝑑 1 and 𝑑 2 , referenced in Eq. ( 7) and Eq. ( 8) respectively. Table 3 illustrates the outcomes of respective schemes. As we can see, the introduction of either 𝑑 1 or 𝑑 2 independently improves our method's performance compared to solely using the model pool M, and further enhancement is achieved when both distance terms are incorporated. This confirms that integrating both distance terms will strengthen the model's performance. It is worth emphasizing that even in the absence of 𝑑 1 and 𝑑 2 , relying solely on the model pool M for training, our approach still outperforms the baselines. This reaffirms the essential paradigm for fostering diversity within the system.4.4.2 Hyperparameter sensitivity. We further investigate the sensitivity of our method to different choices of hyperparameters, as shown in Fig. 7 . The optimal combination of these parameters was then employed in subsequent training. It is evident from the figure that regardless of the adopted search strategies, the performance robustness of our method remains relatively undisturbed across all four examined parameters. This evidence suggests that the performance of our method is robust to the change of hyperparameters.To explore the influence of the order in which domains are trained in domain-shift tasks, we applied our method to the PACS dataset with various domain training orders, as depicted in Table 4 . As we can see, irrespective of the order chosen for domain training, our method consistently surpasses the performance of the baseline methods. This signifies our method's robustness in handling diverse domain order training. 4.4.4 Diversity control measures. Fig. 8 presents the impact of various diversity control measures applied with different distances. Notably, the L2-norm emerges as the best measure, yet, the remaining measures also exceed the performance of baseline methods in most cases. This superiority of the L2-norm is attributed to its ability to measure the true distance between points directly and precisely, offering a more sensitive and effective constraint on point dispersion in the space, thereby yielding superior results.In this section, we visualize the pair-wise distances of the models in the model pool to validate the effectiveness and authenticity of our method. For more case studies, such as the visualization of the classification results of different methods, please refer to our supplementary materials. Fig. 9 illustrates the L2-norm distance matrix for each pair of models within the final client's model pool M 𝑁 , after the completion of training. It is evident that the pairwise distances among all trained models in the model pool display a remarkable variation, without an apparent correlation or trend, such as monotonically increasing/decreasing. This confirms the significant diversity among the models within the pool, attesting to the efficacy of our method in fostering model diversity. By strategically enhancing the disparities between models, we can substantively improve the performance of our approach.For more experimental results, such as the performance of our method on more clients (up to 100 clients), and different data distribution (like Dir(0.1)), more grid search results for the PACS dataset, the impact of different model structures, and PFL adaptation, please refer to our supplementary materials. 
conclusion: 
 In this paper, we highlight the significance of one-shot sequential federated learning for alleviating the communication burdens of current collaborative machine learning paradigms and address the challenge posed by non-IID data. We present a novel one-shot SFL framework with the local model diversity enhancement strategy to reduce communication costs and effectively improve the global model. In particular, we design the local model pool with two regularization terms as a diversity-enhanced mechanism to improve model performance and mitigate the effect of non-IID data. The effectiveness of our method was demonstrated with superior performance on extensive experiments across several datasets including both label-skew and domain-shift tasks. In the future, we will consider integrating more advanced privacy protection measures, adapting to more federated learning settings, and dealing with real-time data under online learning environments to further enhance the feasibility and scalability of the proposed framework. 
acknowledgments: 
 This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-018). 
 
 Appendix:  


Decision: This paper addresses the topic of one-shot federated learning within a sequential federated learning framework. Not all reviewers hold positive stands for this paper.

In my stand, my major concerns include:

- lack of theoretical analysis for the proposed method, like convergence analysis. For federated learning, such analysis is important, and only empirical research is not enough.

- I am not convinced by the experimental setup, which considers only 10 and 4 clients for different datasets. A more commonly used setup is 100 clients in most existing federated learning frameworks.

*** TPC Addendum ***
This paper attracted a diversity of opinions. Given the topic, positive average scores, and increase in scores after rebuttal, the TPC recommends that the conversation continue at the conference with a poster presentation.

Reviews: review: paper_summary: paper_summary: The paper addresses the topic of one-shot federated learning within a sequential federated learning framework. It introduces FedELMY algorithm, which incorporates a Model Pool to tackle the issue of data heterogeneity across clients. Additionally, it introduces a loss function designed to measure the heterogeneity of model parameters within the model pool. The focus on one-shot federated learning is  important as it can significantly enhance communication efficiency. Therefore, I believe the work in this paper is well-motivated. According to the comparison results presented in the paper, the proposed FedELMY algorithm outperforms other baseline methods.
strength_weakness: strength_weakness: 1. The basic idea of the proposed algorithm is easily understandable and intuitive. I especially appreciate the concept of the model pool with the punishing term in the loss function. To my knowledge, this idea is novel and interesting.
2. The paper features great figures and tables, particularly Figure 3 and Algorithm 1, which significantly enhance its readability.
3. The experiments are well-designed, and the results are clearly presented.
4. The availability of open-source code enhances the reproducibility of the experimental results.
limitations: limitations: Here are my concerns or suggestions:
1. Some important content in the section of introduction is described too generally:
    
    1a) Lines 104-106: "Combining multiple networks" is the basic idea of federated learning. This might imply something beyond just the aggregation of models, which should be clarified. The paper uses "networks" to refer to both communication networks and neural networks, which can cause confusion. 

    1b) Lines 108-110: It is unclear why applying such a diversity strategy in SFL is not feasible. More explanation is needed.

2. One of the core parts of the paper is using the proposed loss function in Equation (9) to address the impact of non-IID data. However, there is no theoretical analysis of this new loss function. I recommend using the convergence analysis approach from FedProx, which also integrates a term for dissimilarity in the loss. Without theoretical analysis, it is difficult to understand how hyperparameters affect convergence performance, relying only on experiments with various alpha and beta values.

3. For label skew, it is suggested to use the Dirichlet distribution and validate the performance impact at different levels of non-IID data in terms of label skew.

4. If there are fewer than 10 clients, the experiments may not be convincing, unless it is only designed for small number of clients.

5. When the number of local epochs is equal to 100 or 200, there might be overfitting on the local dataset. This should be clarified.

6. Despite the computational cost, the locally required memory could be a significant issue. Later-selected clients need to cache too many models, which may be unrealistic when the number of clients is high.

7. Formulas (5) and (6) are written in different styles, which should be changed for consistency.
paper_summary: paper_summary: This paper proposed a novel one-shot sequential federated learning training method, called FedELMY, which is designed to solve the non-IID problem. The proposed FedELMY adopts a local model diversity-enhancing strategy for improving model performance. Specifically, FedELMY requires each client to train a model pool for diversity and leverage the averaged model of the model pool as the training output of the client for performance improvement. 
Besides, they introduce two distance regularization terms to enhance diversity and mitigate the negative impact of non-IID data. Experimental results have shown superior performance compared with existing baselines.
strength_weakness: strength_weakness: - The paper as a whole is well-organized, and the figures and graphs throughout the paper are elaborate and clear.
-  A novel framework to improve the global model performance in one-shot sequential federated learning.
- The idea of using the diversity strategy to improve the performance is novel.
- The experimental results are comprehensive and well-grounded.
limitations: limitations: - The topic/motivation of this paper needs to be further polished. In my understanding, this paper focuses on improving the performance of the one-shot sequential federated learning by introducing the diversity strategy. It seems like solving a common problem in a specific area. The non-IID is a common problem in FL, and the paper tries to solve the problem in the new proposed one-shot and sequential setting. It is important to illustrate how hard the non-IID problem is in this new setting, compared with traditional parallel settings, one-shot settings, and sequential settings. In my opinion, the traditional parallel setting and the one-shot setting are sensitive to the non-IID data distribution, while the non-IID has less impact on the sequential setting which can be supported in Table 1 in this manuscript. So, what is the new challenge in solving the non-IID problem by combining the one-shot and sequential? Besides, the main contribution of this paper might be improving the performance of the global model, rather than solving the non-IID problem, since only a regularization term is designed for this. I consider the reason for the performance degradation may lie in introducing the one-shot setting rather than non-IID.

- Motivation aside, the paper tries to improve the performance by introducing the diversity strategy. They design a model pool for diversity, however demanding a higher computational cost for each client. The computation cost brought by the designed FedELMY is important. The paper should provide more detail about the relationship between the computation cost of building the model pool (varying S, rather than local epoch or the number of clients N) and the performance gain. 
- The objective of diversity and mitigating non-IID might be contrasted. As illustrated in Figure 2, the d1 (diversity) and d2 (non-IID impact). To enhance the generalization, averaging a more diverse model is better, which requires a larger d1. However, the significant negative impact brought by non-IID is the local model diversity, that is the d2 should be limited. As designed in Eq.9, I consider the relationship between these two distances (hyperparameters \alpha and \beta) to be the key factor of the performance, which needs to be further explored.
paper_summary: paper_summary: The paper talks about the  challenges in federated learning (FL), specifically focusing on the issues associated with non-IID data distributions in one-shot and sequential federated learning (SFL) paradigms. Traditional federated learning often relies on parallel settings (PFL), which can incur substantial communication and computational costs. To mitigate these costs, one-shot and SFL have been introduced as innovative approaches. However, the non-IID nature of data across clients remains a critical hurdle, limiting the effectiveness of these models due to minimal communication among the clients.

The paper introduces a novel framework called FedELMY (Federated Learning by Enhancing Local Model diversitY), which significantly improves the performance of one-shot sequential federated learning for non-IID data. The key innovation of FedELMY is the enhancement of local model diversity through the creation of a local model pool for each client. This pool includes diverse models generated during local training, which not only captures a broader spectrum of the data distribution but also incorporates two distinct distance measurements to further enhance model diversity and mitigate the effects of non-IID data.
strength_weakness: strength_weakness: 1. Paper Organization:
The paper is well-organized, effectively facilitating a clear and intuitive understanding of the authors' ideas. This structure significantly aids in comprehending the proposed methodologies and their implications.

2. Experimental Setup and Comparisons:
The experimental setup is detailed and the comparisons are thorough. These elements are crucial for validating the effectiveness of the proposed approach and provide a solid foundation for the study's conclusions.

3. Additional Discussions and Code Descriptions:
I appreciate the further discussions and detailed code explanations included in the appendices. These additions are invaluable for replicating the experiments and understanding the finer details of the implementation.

4. Central Idea of the Paper:
The central idea of the paper is both straightforward and intriguing. It proposes generating a range of diverse models to effectively address the challenges posed by Non-IID Data in Sequential Federated Learning. This innovative approach has the potential to make a contribution to the field.
limitations: limitations: Thank you very much for your submission and for trusting the multimedia community. The following contents are not so much limitations of the paper as points for discussion.

1. **Curiosity about the choice of distance parameters d1 and d2**: What is the reason for choosing the L2 distance for the distance parameters d1 and d2 in this paper? Why not consider other types of distance metrics?

2. **Regarding the setting of the hyperparameter $\alpha$**: If I have overlooked something, please remind me. I do not seem to have seen the settings for the hyperparameter $\alpha$ mentioned in the paper, as the section 4.1 Experimental Setup only seems to mention $\beta = 0.5$.

3. **Interest in the settings of the two hyperparameters**: The settings of these two hyperparameters are interesting because they respectively require the model to distance itself from previous models and to stay close to the initial model. Therefore, I speculate that one of the constraints might become ineffective depending on the settings of these parameters. However, I am puzzled by the conclusions in Figure 7(c) and Figure 7(d). If this model is not sensitive to these two hyperparameters, does it imply that setting them to zero would not significantly impact the performance of the algorithm?

4. **Regarding Table 3**: It appears that the performance improvement of the algorithm mainly comes from the model pool (about a 5-point increase), while the contributions of d1 and d2 seem quite limited (around 1%). Furthermore, considering Figure 7, when $\alpha$ is set to less than 0.06 and $\beta$ to 0.6 or 0.8, the performance of the algorithm appears to be inferior to just using the model pool version. The analysis in Figure 7 indicates that the algorithm is robust to the performance variations with $\alpha$ and $\beta$, but it seems that different settings can even lead to negative outcomes.

In summary, I find the idea of the paper interesting, but I am unable to determine the effectiveness of d1 and d2 through either theoretical reasoning or experimental results. Therefore, I believe this paper might require further discussion.