One-shot-but-not-degraded Federated Learning

Authors:   client
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 05:08:51

Abstract: Transforming the multi-round vanilla Federated Learning (FL) into one-shot FL (OFL) significantly reduces the communication burden and makes a big leap toward practical deployment. However, we note that existing OFL methods all build on model lossy reconstruction (i.e., aggregating while partially discarding local knowledge in clients’ models), which attains one-shot at the cost of degraded inference performance. By identifying the root cause of stressing too much on finding a one-fit-all model, this work proposes a novel one-shot FL framework by embodying each local model as an independent expert and leveraging a Mixture-of-Experts network to maintain all local knowledge intact. A dedicated self-supervised training process is designed to tune the network, where the sample generation is guided by approximating underlying distributions of local data and making distinct predictions among experts. Notably, the framework also fuels FL with flexible, data-free aggregation and heterogeneity tolerance. Experiments on 4 datasets show that the proposed framework maintains the one-shot efficiency, facilitates superior performance compared with 8 OFL baselines (+5.54% on CIFAR-10), and even attains over $\times$4 performance gain compared with 3 multi-round FL methods, while only requiring less than 85% trainable parameters.

Main paper: 
 related work 2.1 one-shot federated learning: 
 One-shot FL is a variant of the FL that requires only one communication round between the server and clients. In OFL, the clients upload well-trained local models to the server, and the server aggregates these uploaded models to obtain a new global model [12] . Existing methods in OFL can be categorized into knowledge distillationbased and parameter optimization-based methods.2.1.1 Knowledge Distillation-based. These methods are proposed to use knowledge distillation (KD) to transfer the massive local knowledge into one global model. They use the local models as teachers, and use the auxiliary public dataset or model to train a student model. [12] firstly proposed to use the KD and use the ensemble prediction of local models as the teacher's output. FedKT [22] designs a two-tier PATE structure [30] relying on public data to improve the ensemble of local models. To alleviate the label skews, FedOV [9] adopts open-set voting in OFL to enhance the generalization ability of the ensemble. However, these methods require the auxiliary public data or pre-trained models should be similar to the original task, which limits the application in real-world scenarios, especially in privacy-sensitive scenarios such as biomedical domains, since there might be no available data or models.To this end, some methods try to utilize additional distilled data or synthetic data instead of public data. [49] proposes transmitting a distilled dataset to the server. DENSE [47] firstly proposes a data-free OFL method through training a data generator to assist the KD process. Co-Boosting [7] designs a mutually enhanced process to synthesize high-quality samples and distillation models. Furthermore, FedCAVE [16] modifies the local learning task into training a conditional variation auto-encoder (CAVE) and uses KD to compress the ensemble into a powerful decoder. The decoder can be used to generate training samples for the global model. And the FedCADO [40] adopts the popular diffusion models to get the synthetic data. However, none of the aforementioned methods can fully use the knowledge of local models, since the KD methods have been criticized as inefficient [48] , and existing an intractable performance gap.2.1.2 Parameter optimization-based. These methods aim to search for an optimum across all local models by analyzing the local model parameters. Clearly, the traditional statistic aggregation methods, such as FedAvg, Median [42] , and Krum [3] can still apply in OFL but achieve low performance. k-FED [8] runs a variant of Lloyd's method for k-means clustering and obtains an aggregated model through one round iteration of exchanging local cluster means. MA-Echo [35] tries to get the Pareto optimum of the local clients via exploring common harmonized optima. However, directly analyzing the model parameters requires all local models should be homogeneous, whose setting is not practical in real-world heterogeneous scenarios. And the optimum of the parameters can not represent the optimum of the model performance, in most cases, the performance degrades obviously [47] . Besides, none of these methods support the model heterogeneity, i.e., different clients have different model architectures [23] . It is challenging to get the optimum from completely different architecture parameters.The MoE [6, 18] is an ensemble learning framework that combines multiple expert networks to enhance the overall performance of a model. Each expert in an MoE model is specialized in handling specific types of data or tasks. A gating network, which is a crucial component of MoE, dynamically routes input tokens to the appropriate experts, ensuring that only relevant parts of the model are activated for a given task. Based on this, several improvements have been proposed to reduce the training cost [14, 32] , or improve performance on multi-tasks [6, 28] .Recently, MoE has become promising again due to the appearance of Mixtral8×7B, which uses the MoE for model scaling and ... achieves better performance than a larger one. Moreover, GShard [20] uses MoE for scaling up large models and managing models effectively. Switch Transformers [11] introduces a model scaling method by utilizing the MoE to substitute the FNN in Transformer, achieving better training efficiency and model performance. MoE architecture can also be applied to solve the multi-task or multi-modal problem. LLaMA-MoE [36] build a series of MoE models based on LLaMA [37] with continual pre-training. MoE-LLaVA [25] adopt this architecture to achieve high performance and fewer parameters in Large Vision-Language Models (LVLM).Existing methods also introduce the MoE into FL, and all these methods focus on solving the heterogeneity problem in personalized FL [39] . PFL-MoE [13] views the global model and personalized local model as two experts and utilizes the MoE to achieve better personalized performance and generalization. FedMoE [41] tries to solve the model heterogeneity problem and uses the MoE to weigh the representations of the global homogeneous model and local personalized heterogeneous models. However, these methods still adopt multi-round averaging, which suffers from performance degradation as the parameter optimization-based OFL methods. Contrarily, we adopt the MoE in OFL aggregation, focusing on better leveraging the knowledge of local models.  
formulation and design of intactofl 3.1 learning problem and goal: 
 EQUATIONwhere ℓ represents the loss function corresponding to the OFL task, for example, ℓ can be the cross-entropy function in the classification task. 𝑓 (𝑥 𝑖 ; 𝑤 𝑔 ) is the prediction function that output the prediction of 𝑥 𝑖 when given the model parameter 𝑤 𝑔 . In OFL, the server can not directly access the data D = ∪ 𝑚 𝑘=1 D 𝑘 , and only well-trained local models 𝑤 𝑘 are accessed. To facilitate this, existing methods mostly can be categorized into knowledge distillation-based and parameter optimization-based methods.For knowledge distillation-based methods, the well-trained local models are integrated into an ensemble, and the ensemble model acts as teacher to guide the global model (student). The ensemble model can be defined as follows,EQUATIONwhere 𝑓 (𝑥; 𝑤 𝑘 ) is the output of the input sample 𝑥 given the local model 𝑤 𝑘 , while 𝛽 𝑘 is the weight of client 𝑐 𝑘 . In most settings, the 𝛽 𝑘 is assigned to𝛽 𝑘 = 1 𝑚 or 𝛽 𝑘 = 𝑚 𝑘 / 𝑚 𝑗=1 𝑚 𝑗 .Relying on the public auxiliary data or synthetic data, i.e., D 𝐴 , the distillation process can be defined as follows:EQUATIONwhere the ℓ can be the Kullback-Leibler (KL) divergence function corresponding to the distillation task. Notably, existing literature has reported that the performance gap of knowledge distillation [39, 50] , the key part is the distillation distance between the ensemble and global model. ). The objective of these methods can be written as: minEQUATIONwhere A ({𝑤 𝑘 } 𝑚 𝑘=1 , 𝑤 𝑔 ) can be any distance functions or similarity functions. WhenA ({𝑤 𝑘 } 𝑚 𝑘=1 , 𝑤 𝑔 ) = ∥ 𝑚 𝑘=1 𝜌 𝑘 𝑤 𝑘 -𝑤 𝑔 ∥ 2 , the global model is 𝑤 𝑔 = 𝑚𝑘=1 𝜌 𝑘 𝑤 𝑘 , which is the same as the vanilla aggregation algorithms FedAvg in a one-shot manner. However, these methods focus on processing the model parameter, which is far away from the original OFL objective.In IntactOFL, we aim to preserve all local models' knowledge through direct integration with MoE. We define the objective of IntactOFL as:EQUATIONwhere 𝑓 M (𝑥 𝑖 ; {𝑤 𝑘 } 𝑚 𝑘=1 ) is the output of the MoE network. The goal is to fully utilize all local models and form an MoE network M which can achieve high performance on predefined tasks. For the MoE network M, it consists of gating network G and a set of experts 𝐸 = {𝑤 𝑘 } 𝑚 𝑘=1 . The MoE architecture dynamically adapts the weight of each expert 𝜌 𝑘 through the gating network G and maximizes the influence of all experts. The weight can be described as:EQUATIONFor any input sample, the gating network G will dynamically adjust the weights of the experts and distribute these samples to specialized experts for high performance. We use the weighted outputs of the experts as the final output of the MoE network. To this end, the 𝑓 M (•) can be formulated as:EQUATIONTherefore, the objective of IntactOFL can be rewritten asEQUATIONThe illustration of IntactOFL is shown in Figure 2 . After the clients upload their well-pre-trained models, the server aggregates these local models with an MoE architecture. The one-shot MoE network consists of the experts and a gating network. The server treats all the local models as the experts and trains a gating network in a self-supervised manner. Specifically, each training epoch consists of two steps: (1) the current one-shot MoE network guides the data generation process of the Generator 𝑔 with L 𝑡 𝑔𝑒𝑛 , given random noise, the 𝑔 can produce many auxiliary samples with similarity and informativeness. (2) Then we add the noise on these samples for data augmentation. These generated samples are treated as the input of the one-shot MoE network. We adopt the L 𝑡 𝑡𝑟𝑎𝑖𝑛 to update the gating network for forming a better MoE network. Note that the self-supervised MoE training consists of two major processes, which are self-supervised data generation and one-shot MoE network training. The self-supervised data generation is designed for data-free training, and the latter one is for flexible aggregation. We present the details of these two processes in § 3.3 and § 3.4, respectively.Data generation fuels the self-supervised process with optimization basics. For each specific pseudo label, the generator is expected to render noises as the samples with this label. Initially, the MoE's prediction on a generated sample would be very different from the expected pseudo label, which would lead to the optimization of the generator and, more importantly, the gating network before the MoE, via back-propagation. This self-tuning process proceeds until the generator learns how to generate and the gating network learns how to assign weights.Similarity Measure. Basically, the generated samples are settled to mimic the raw local data by representing a similar distribution. We consider the similarity between the generated auxiliary data and the training data. Note that the similarity only focuses on the similar data distribution for utility, rather than visual reconstruction. Following the existing data-free methods, we adopt the Cross-Entropy loss function [7, 47] . Different from existing methods which adopt statistic ensemble output to guide the data generator training, we dynamically use the updated MoE network. The loss for similarity in training epoch 𝑡 can be formulated as:EQUATIONwhere x is the auxiliary data with a corresponding label 𝑦. Informativeness Measure. Optimized only with the CE loss, the generated auxiliary data can be easily fitted, while may be less representative and contains less information for fine-tuning the backbone gating network. Inspired by [7, 10] , the hard samples can embed more representative information compared with easy-tofit samples. In our context, the samples resulting in more diverse outputs across different experts are considered harder to fit (i.e., more informative in optimizing the gating network). Hence, we use the variance of the experts' output, i.e., 𝜎 (𝑓 𝑡 M ( x)), as the informativeness of each sample. With this, we could explicitly increase the preference for hard samples in data generation.By jointly considering similarity and informativeness, the loss function that guides data generator training is formulated as:EQUATIONwhere 𝜎 (•) is the variance function.To further promote sample diversity, we introduce a data augmentation module after data generation. Specifically, we further add noise back to the originally generated sample following [7] . As such, we can get more diverse auxiliary samples D 𝐴 of which similarity and informativeness.The goal of training the MoE network is to make the best of the knowledge of local data. We use the generated auxiliary data as the training data. To preserve the knowledge of local models without disruption, we froze all experts, since the experts' updating would cause unavoidable forgetting [2] . Instead, we focus on training a high-performance gating network to make the best use of G ← G -𝜂 G ∇ G L 𝑡𝑟𝑎𝑖𝑛 (G), according to Eq.( 13).13:end for 14: end for the knowledge of local models. To this end, we update the gating network according to Eq. (11) .min G L M ({𝑤 𝑘 } 𝑚 𝑘=1 ) = 1 |D 𝐴 | ∑︁ (𝑥 𝑖 ,𝑦 𝑖 ) ∈D 𝐴 ℓ ( 𝑚 ∑︁ 𝑘=1 𝜌 𝑘 (𝑥; G) 𝑓 (𝑥; 𝑤 𝑘 ), 𝑦 𝑖 ).(11) Meanwhile, the 'over-trust' problem of the gating network has been widely reported [20, 33] , that is, the gating network always produces large weights for the same few experts. Following [33] , we add a balance loss to mitigate the weight bias of the gating network. Specifically, we compute the importance of each expert in every batch in 𝑡 iteration, and the balance loss equals the square of the coefficient of variation (CV) of the set of importance (see Eq.( 12)).𝜆 is the scaling factor for the balance loss, which is set as 1 in the implementations.EQUATIONIn summary, the MoE training loss function consists of two parts, the first part is for high performance which minimizes the difference between model outputs and labels; the second part is for preventing the "over-trust" phenomenon and achieving a balanced gating network. We conclude the loss function as:EQUATION)The details of the training process are presented in Algorithm 1. 
evaluation 4.1 experiment setup: 
 We provide the important details of the datasets, data partition, and baselines here, and present the rest details in the Appendix.4.1.1 Datasets. We evaluate the proposed IntactOFL and baselines on four widely used classification benchmarks: CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet. CIFAR-10 consists of 60,000 32 × 32 RGB images in 10 categories. It has 50,000 training samples and 10,000 test samples. CIFAR-100 has the same format as CIFAR-10, except it has 100 categories. SVHN is a real-world dataset comprising over 600,000 digit images extracted from Google Street View images. Tiny-ImageNet is the subset of the ImageNet dataset, containing 200 categories and 100,000 images (500 images for each category), each image is downsized to 64 × 64.To simulate the real-world heterogeneous environment, we adopt the Dirichlet distribution (𝛼) to control the proportions of each category across clients [21, 44] . We use the 𝛼 to control the degree of heterogeneity. A small 𝛼 represents a biased data distribution. Following the settings in [47] , we set 𝛼 = {0.05, 0.1, 0.3, 0.5}.We compare the performance of the proposed IntactOFL against the existing two categories of methods. One is the knowledge distillation-based method, another is the parameter optimization-based method. We compare with knowledge distillation-based SoTA which are DENSE [47] and Co-Boosting [7] . Meanwhile, following the setting in [7, 47] , we also compare with the FedDF [26] which uses the real validation dataset for distillation, and some baselines derived from prevailing data-free knowledge distillation methods which are F-ADI [43] and F-DAFL [4] . For parameter optimization-based methods, we choose high-performance MA-Echo [35] and vanilla FedAvg in a one-shot manner [12, 29] . To ensure fair comparisons, we omit the comparison with methods that require multi-round interactions, such as FedProx [23] , SCAFFOLD [19] , and FedCav [45] . We also neglect some methods that require additional public data or models, such as FedKT [22] , FedOV [9] and FedGen [50] . 1 shows the effectiveness of the proposed IntactOFL, we conduct experiments under various data heterogeneity settings across different datasets and methods by vary 𝛼 = {0.05, 0.1, 0.3, 0.5}. From the table, we conclude that the proposed IntactOFL is effective in data heterogeneity settings. Specifically, (a) In all data heterogeneity settings, the IntactOFL outperforms than baselines. The IntactOFL surpasses the best baseline by substantial margins with 5.54%, 4.26%, 3.50%, and 2.82% on CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet, respectively. Even in a more heterogeneous environment (𝛼 = 0.05), IntactOFL achieves over a 5% accuracy improvement compared to the best baseline. (b) Notably, the knowledge distillation-based baselines achieve better performance than the parameter optimization-based methods. The DENSE and Co-Boosting both achieve ∼3% test accuracy higher than MA-Echo and FedAvg. The reason is that the parameter reconstruction process of optimization-based methods does not preserve the local models' knowledge, and the knowledge distillation-based methods can transfer this knowledge to a new model. Owing to the MoE architecture which preserves the local models, the proposed IntactOFL can achieve better performance among these baselines. (c) Besides, the Ensemble which equally averages all local models' output can achieve the second-best performance in some settings. This is thanks to the benefits of preserving all information. However, the compromised results obtained by equally averaging limits its performance in all settings. The proposed IntactOFL adopts the gating network for effective utilization of local models' knowledge evidently improves the entire performance. In summary, the In-tactOFL is effective in various data heterogeneity scenarios and achieves competitive performance than baselines.We evaluate the scalability of IntactOFL in two aspects: horizontal scalability, which evaluates the performance in a larger distributed network with more clients; and vertical scalability, which evaluates the performance in scenarios of model heterogeneity among clients, where different clients can possess entirely distinct model architectures.For horizontal scalability, we evaluate the test accuracy of all methods in diverse networks by varying the number of clients 𝑚 = {5, 10, 25, 50, 100} on CIFAR-10 and SVHN. Table 2 shows the results of different methods across different clients 𝑚 in CIFAR-10, the results of SVHN are presented in the Appendix. As suggested in [24] , the server can become a major bottleneck while the number of clients increases. We also reach a similar conclusion, with the number of clients 𝑚 increasing, the performance decreases, which is consistent with [7, 24, 47] . Even though the negative impact on the accuracy, our methods still achieve better performance than other baselines, which also verifies that the knowledge-preserved method brings a better aggregated global model. In summary, the IntactOFL is scalable across diverse distributed networks of varying sizes.For vertical scalability, we note that our proposed methods can support heterogeneous models, which are scalable to different model architectures. We set five different model architectures from simple to complex, that is, MLP, CNN, MobileNetV2 (Mob) [31] ,ResNet (Res) [15] , and VGG [34] . We evaluate the test accuracy of all methods with the same setting in § 4.2.1 except 𝛼 = 0.5. Besides, we consider a challenging setting in FL, in which both the data and models are heterogeneous across all clients. Different clients can train their own models on their local data with entirely different model architectures. We remark that this practical setting is complicated in existing methods [7, 17, 47] . Thus, we conduct two types of evaluations to verify the ability to support heterogeneous models. Since some baselines do not support the model heterogeneity, we omit these methods and only report the Ensemble, DENSE, and Co-Boosting.For the first type, we evaluate the performance by varying the model architectures of entire systems. All clients are homogeneous and test on different model architectures. Results are demonstrated on the left panel of Figure 3 . We conclude that the model architecture can impact the performance, a model with better representation ability can achieve better performance. The proposed method achieves the best performance across all model architectures.The second type concerns a scenario where the model architectures among clients are completely heterogeneous. We set three cases of these complete model heterogeneity, see the central panel in Figure 3 . For example, the Case #1 represents that Client 0 uses MLP to train a local model with the test accuracy of 39.92%, after the local training, the server would collect the MLP from Client 0, VGG from Client 1, ResNet from Client 2, CNN from Client 3, and MobileNet from Client 4. We report the test accuracy of the global model, which is aggregated by completely heterogeneous local models. The results (see the right panel in Figure 3 ) demonstrate that even in complete model heterogeneity settings, our methods can achieve ∼10% accuracy improvement than existing baselines. In summary, our proposed method is heterogeneity-tolerant and significantly outperforms other baselines. In summary, the proposed IntactOFL is scalable both in large distributed network and model heterogeneity scenarios and outperforms other baselines by a large margin. 4.2.3 Efficiency. Note that the proposed IntactOFL is efficient. We evaluate the efficiency from two aspects: parameter efficiency and computation efficiency. The parameter efficiency means the required parameters should be as small as possible. The computation efficiency means the training iterations should be as few as possible. We adopt the same setting as the § 4.2.1.For parameter efficiency, we report the # trainable parameters and the test accuracy of DENSE and Co-Boosting across different global model architectures. The trainable parameters of DENSE and Co-Boosting mainly include the data generator and student model for distillation. In IntactOFL, there is no additional global model, the trainable part only consists of the generator and lightweight gating network. As shown in Figure 4 , thanks to the lightweight gating network, the proposed IntactOFL achieves the highest accuracy while requiring the smallest trainable parameters (85% of DENSE(CNN)) on the server.Notably, the IntactOFL is also computation-efficient. We consider that the computation of the local training is essential to learn the local knowledge. However, the aggregation process is inefficient, so existing methods introduce additional computation overhead to distill local models or require multi-rounds of local training. To better evaluate the computation overhead, we design a new factor named Performance Gain (PG), which can be formulated as:EQUATION)where 𝑃 is the performance, and in the classification task, 𝑃 is the test accuracy; 𝐶 𝑠𝑒𝑟 𝑣𝑒𝑟 is the computation cost on the server, including training the generator, distilling, and so on. 𝐶 𝑐𝑙𝑖𝑒𝑛𝑡𝑠 is the sum of local training iterations of all clients. Here, We use a coarsegrained estimation of the number of iterations 𝑇 to approximate the computational cost, which is 𝐶 𝑠𝑒𝑟 𝑣𝑒𝑟 ≈ 𝑇 𝑠𝑒𝑟 𝑣𝑒𝑟 and 𝐶 𝑐𝑙𝑖𝑒𝑛𝑡𝑠 ≈ 𝑇 𝑚 × 𝑚 𝑘=1 𝐸 𝑖 , where 𝑇 𝑚 is the interaction rounds between the server and clients. For OFL, 𝑇 𝑚 = 1, for multi-rounds FL, 𝑇 𝑚 > 1. We use the PG to evaluate how much the performance gain is brought by each training iteration.Furthermore, we introduce some representative multi-rounds federated learning methods for comparison, which are multi-rounds FedAvg (M-FedAvg) [29] , SCAFFOLD [19] , and FedCav [45] . As reported in Table 3 , we achieve the highest 𝑃𝐺 across all baselines, which means that our methods can obtain the highest global model accuracy improvement for each training step. It is worth noting that, even though the multi-rounds FL algorithms can achieve the best performance, its 𝑃𝐺 is significantly less than that of the OFL, which also verifies the inefficiency of the existing multi-rounds aggregation methods [47] . In summary, the proposed IntactOFL is parameter-efficient and computation-efficient compared with other baselines, which requires less trainable parameters and fewer computation iterations while achieving higher performance.In this part, we investigate the impact of different MoE architectures. Since the clients determine the local models' (experts) architecture, we focus on the performance by varying different gating networks. Softmax gating, noisy Top-K gating [32] , and MMoE [28] are three popular gating networks in MoE. The Softmax gating uses the softmax function to normalize the weights across all experts. The noisy Top-K gating adds noise N ∼ (0, 𝜎) to the original weights and only selects top k experts. MMoE adopts multiple gating networks for multi-tasks and here we derive this method by averaging the outputs of these 𝐾 𝑔 gating networks as the final output. Here, we use the averaged memory consumption of one batch of samples to represent the computational cost. We test these methods on the CIFAR-10 with CNN among five clients. We report the performance and memory consumption of different gating networks in Figure 5 . (a) We notice that the test accuracy and memory consumption act as a trade-off, utilizing more experts can achieve high performance meanwhile 00R(K g = 2 00R(K g = 3 00R(K g = 5 4.3.2 Impact of Generator. We provide the visualization of the generated data on CIFAR-10 and SVHN in the Appendix. Note that the auxiliary data are generated for utilization rather than visual reconstruction. Besides, we investigate the impact of the key components in data generation. We conduct leave-one-out testing and report the results by removing data augmentation (w/o Aug), removing informativeness (w/o Inf ), and removing both (w/ Sim).7RS = 0 7RS = 0 7RS = 0 7RS = 0 7RS = 1 7RS = 1 7RS = 1 7RS = 1As shown in Table 4 , only considering the similarity leads to poor performance, which is consistent with [7, 47] . The informativeness module is essential for performance improvement compared with the data augmentation. A combination of these components leads to a high performance, which shows that all these components have contributed to performance improvement. 
conclusion: 
 In this paper, we aimed to bridge the knowledge loss of existing one-shot federated learning methods, where the model reconstruction process results in significant performance degradation. We proposed IntactOFL, a novel method that adopts the Mixture of Experts' architecture to preserve all local models' knowledge, achieving high performance through dynamic weighting by a gating network. We designed a self-supervised MoE training framework by iteratively generating samples and updating the gating network. Extensive experiments have verified the effectiveness, scalability, and efficiency of the proposed IntactOFL. 
: 
 As a remedy, one-shot FL (OFL) has emerged recently by reducing the interaction of vanilla FL to just one communication round [12] . In fact, OFL is believed to be well-suited for the popular model market scenarios [47] , where users are willing to sell local-trained models but are reluctant to join in the redundant training process. Technically, existing designs of OFL, now in its infant age, could be categorized into knowledge distillation methods (KD) [16, 22, 40, 47, 49] and parameter optimization methods [8, 35] , both works in a way of aggregating all local models into one global model. Wherein, KD methods transfer knowledge from local models to a global model with assumed-to-be-available auxiliary data, while the latter one focuses on designing dedicated averaging [29, 45] , clustering [8] , and solving Pareto optimum [35] for one-shot.Unfortunately, the one-shot gain of existing OFL methods comes at the cost of degraded learning performance. The above proposals all try to learn a one-fit-all global model given local models, which is essentially a model reconstruction process, accompanied by knowledge loss. The visualization in Figure 1 shows the differences in correct-prediction domains (i.e., samples that are correctly predicted) between local models and existing OFL models. Notably, both knowledge distillation and parameter optimization methods reconstruct the knowledge space in the union of already-learned local knowledge, while large sub-areas (i.e., those not covered after OFL) are discarded during aggregation. As a result, samples falling in the discarded sub-areas would experience wrong inference with existing OFL models.The FL paradigm would be more fantastic if the one-shot is attained without performance degradation. For this purpose, this work presents the novel design Intact One-shot Federated Learning (IntactOFL), with the key idea of synergizing the knowledge of local models rather than reconstructing it. By regarding each local model as an expert, IntactOFL employs the Mixture of Experts (MoE) [18] architecture to retain all the local knowledge, where a gating network is planted for flexible expert/local model assignment during inference (i.e., having which local models to make prediction on a specific sample), thus relieving the one model-fit-all pitfall. With the expert model frozen, the gating network, as the core in our one-shot MoE network, is trained in a self-supervised way based on auxiliary sample generation. Specifically, a generator is designed to construct samples that, on one hand, mimic the local data with the pseudo label approximating the prediction of local experts, and on the other hand, are hard to predict by the MoE network and thus fuel the optimization of it.We elaborate on and highlight three advantages of IntactOFL: First, in contrast to the previous OFL's reconstruction-based design, IntactOFL preserves all local models' training efforts and utilizes a lightweight gating network to essentially construct a dedicated model for each sample (i.e., flexible aggregation). Second, it requires no additional data or pre-trained models (i.e., data-free), which well-suits the privacy-sensitive scenarios of FL, especially compared to existing KD methods. Third, by focusing on the fusion of knowledge rather than the aggregation of models, IntactOFL could by nature handle heterogeneous clients with non-iid data distribution and heterogeneous model architectures (i.e., heterogeneity tolerance).Overall, our main contributions can be summarized as follows:• We first identify the knowledge loss pitfall of existing OFL methods, incurred by the reconstruction of the global model from local models and appeal to synergizing knowledge instead. We believe this could spur rethinking the default way of performing aggregation in FL.  
 
 Appendix:  


Decision: This work aims to improve the model's performance under the one-shot federated learning scenario. 

Most reviewers acknowledge the originality of the work. Before the rebuttal, the reviewers had concerns about unclear experimental results, unclear claims, and expensive computation costs. After the rebuttal, all reviewers acknowledged that their concerns had been solved.

Currently, the ratings of all reviewers are positive. Therefore, the AC would like to accept this paper. The authors are suggested to carefully revise the paper based on the concerns and suggestions from reviewers in the camera-ready version.

Reviews: review: paper_summary: paper_summary: Summary:
This paper studies the one-shot federated learning problem where the communication between server and clients is only conducted one time. Due to existing methodologies that aggregate while partially discarding local knowledge in clients’ models would lead to significant performance degradation, the authors claim it is caused by stressing
too much on finding a one-fit-all model. In order to solve this problem, the authors propose to conduct a mixture-of-experts mechanism to guarantee all local knowledge stays intact, thus avoiding performance degradation in one-shot federated learning. Moreover, the authors propose self-supervised training by generating data guided by approximating underlying distributions of local data and making distinct predictions among experts. Once the self-supervised training is done on the gating network, a one-shot MoE training is conducted to guide the learning of each expert.
strength_weakness: strength_weakness: Strengths:
* The motivation is clear and reasonable.
* This paper is well-organized and easy to follow.
* The experimental results are good, and the evaluation is extensive and sufficient.
limitations: limitations: Weaknesses:
* How did Figure 1 is generated? If it is manually plotted, the conclusion of “local models are discarded with knowledge distillation and parameter optimization” is less convincing.
* Why employing self-supervised learning is important for the one-shot federated learning problem? It seems that the motivation for employing data generation to boost learning performance is orthogonal to the motivation of this paper. Could there be any justification? If self-supervised training is important, how does this module influence general performance? Moreover, what does the generated data look like? Are there any visualizations?
* It seems that the proposed method still has slight performance degradation than SCAFFOLD, is there any explanation?
* The similarity and informativeness are two other incremental contributions. For a fair comparison, it is suggested to employ these two modules to a typical baseline method and then compare the performance.
* The datasets for evaluation are relatively small-scale, it is suggested to incorporate several typical larger federated learning benchmarks.
* How efficient is the proposed method? It is suggested to conduct a theoretical or experimental evaluation to further justify the efficiency of the proposed method compared to typical federated learning baseline methods, such as SCAFFOLD.
* Federated learning under OOD scenarios is a very practical research problem, how does the proposed method perform under OOD datasets? Could the authors provide several insights or justification?
paper_summary: paper_summary: This work aims to promote the model's performance under the one-shot FL scenario. To this end, the authors propose a novel strategy to leverage a MioE network to maintain knowledge from local clients. Some experiments are conducted to verify the effectiveness of the proposed method.
strength_weakness: strength_weakness: The proposed method is technically sound. Moreover, leveraging self-supervised data generation is novel in one-shot FL scenarios.

The performance gain is decent. The ablation studies conducted provide a clear picture of the proposed method's effectiveness.
limitations: limitations: The experimental results are unconvincing and require clear clarification. Specifically, the authors compare the proposed method and M-FedAvg with $\alpha=0.5$, aiming to verify the point "not-degraded". However, the performance of M-FedAvg does not align with previous works. For instance, M-FedAvg can achieve good test accuracy when $\alpha=0.5$, e.g., $>85\%$. This significantly weakens the contribution of this work. The authors must claim or clarify that the code base is the same for all experiments.
paper_summary: paper_summary: Existing OFL approaches all build on model lossy reconstruction, which attains one-shot at the cost of degraded inference performance.
To address this issue, this paper proposes a novel one-shot FL framework by embodying each local models as an independent expert 
and leveraging a MoE network to maintain all local knowledge intact. A dedicated self-supervised training process is designed to train the MoE network, where the sample generation is guided by approximating underlying distributions of local data and making distinct predictions among experts. Extensive experiments demonstrate the superiority of the IntactOFL method.
strength_weakness: strength_weakness: 1. The motivation for this work is clear.
2. Unlike the common knowledge distillation methods and parameter optimization methods, the authors use the MoE network for OFL. 
And treat each client as an expert.
3. The authors utilized auxiliary data generated by the generator to train the MoE network.
4. The authors have given enough consideration to experiments, and the paper has a relatively large number of types of experiments.
limitations: limitations: 1. The authors claim that IntactOFL is compared to 8 OFL baselines. But it is always known that vanilla FedAvg is a method that requires
multi-round interactions, not a OFL method.

2. Whether it would be better to train Gating network directly using proxy datasets (some KD methods also utilize proxy datasets), 
 I'm not sure if the authors have considered.

> 1.Yi L, Yu H, Ren C, et al. FedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning[J]. arXiv preprint arXiv:2402.01350, 2024.

> 2.Parsaeefard S, Etesami S E, Garcia A L. Robust federated learning by mixture of experts[J]. arXiv preprint arXiv:2104.11700, 2021.

> 3.Guo B, Mei Y, Xiao D, et al. PFL-MoE: Personalized federated learning based on mixture of experts[C]//Web and Big Data: 5th International Joint Conference, APWeb-WAIM 2021, Guangzhou, China, August 23–25, 2021, Proceedings, Part I 5. Springer International Publishing, 2021: 480-486.
paper_summary: paper_summary: This paper introduces a novel one-shot federated learning algorithm, IntactOFL. The algorithm uses a MoE network to maintain the integrity of all local knowledge. By using the gating network of the MoE to integrate local models and alternating the training of an auxiliary data generator and the gating network, it achieves one-shot federated learning through the generation of auxiliary data. Experimental results demonstrate that this federated learning approach achieves one-shot communication without any performance loss and even outperforms traditional federated learning methods.
strength_weakness: strength_weakness: The proposed approach, IntactOFL, is bold and innovative. I like the idea of integrating a synthetic data generator into MoE for training a global model. The paper is well-written and clear in both its problem formulation and algorithm description. Additionally, through experimental study, IntactOFL outperforms other baselines on various datasets, validating the effectiveness of one-shot federated learning in IntactOFL without any performance loss. The visualizations of the auxiliary data points in the supplementary material demonstrate the privacy capabilities of IntactOFL. Furthermore, the approach can be applied to scenarios where the local models across clients differ.
limitations: limitations: 1. Figure 1 lacks sufficient insight and could be improved to better illustrate the concept. The statement in Figure 1 should be clarified with additional supporting analysis.

2. I am curious about the optimal size of the generated auxiliary data for achieving good performance. How large should the auxiliary dataset be? The size of the auxiliary dataset will impact communication costs. Although one-shot federated learning only requires one round communication, the single communication load for transmitting the auxiliary dataset is significant. Additionally, the size of the auxiliary dataset could affect the final prediction results. Therefore, a detailed discussion and guidelines on the optimal size of the auxiliary data are necessary.

3. The IntactOFL method proposed in the paper seems to require significant computational resources. The process involves training the auxiliary data generator and the gating network successively in a loop, which may demand extensive computational power. Along with the discussion of this limitation, the paper should provide some potential optimization methods or directions to address this issue. Otherwise, the proposed one-shot federated learning method could be limited in edge scenarios with restricted computational capabilities.

4. Some minor points:

    4a) More categories of one-shot federated learning are missing here, such as Dataset-distillation-based one-shot federated learning.

    4b) The visualization of auxiliary data (now in supplementary material) should be better included in the main paper, as the generated data for the gating network is a crucial part of the proposed framework.

    4c) Although it might be somewhat difficult, providing more theoretical analysis regarding the convergence of the proposed training process would be beneficial.

    4d) It is recommended to provide open-source code for the replication of the experimental results.

    4e) The notations $m$ and $m_k$ are a bit confusing. I suggest using different symbols to distinguish between the number of clients and the number of data points.