DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices

Authors: yongzhe  jia
xuyun  zhang
hongsheng  hu
kim-kwang raymond choo
lianyong  qi
xiaolong  xu
amin  beheshti
wanchun  dou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:09:33

Abstract: Federated learning (FL) has emerged as a prominent machine learning paradigm in edge computing environments, enabling edge devices to collaboratively optimize a global model without sharing their private data. However, existing FL frameworks suffer from efficacy deterioration due to the system heterogeneity inherent in edge computing, especially in the presence of domain shifts across local data. 
In this paper, we propose a heterogeneous FL framework DapperFL, to enhance model performance across multiple domains. In DapperFL, we introduce a dedicated Model Fusion Pruning (MFP) module to produce personalized compact local models for clients to address the system heterogeneity challenges. The MFP module prunes local models with fused knowledge obtained from both local and remaining domains, ensuring robustness to domain shifts. Additionally, we design a Domain Adaptive Regularization (DAR) module to further improve the overall performance of DapperFL. The DAR module employs regularization generated by the pruned model, aiming to learn robust representations across domains. Furthermore, we introduce a specific aggregation algorithm for aggregating heterogeneous local models with tailored architectures and weights. We implement DapperFL on a real-world FL platform with heterogeneous clients. Experimental results on benchmark datasets with multiple domains demonstrate that DapperFL outperforms several state-of-the-art FL frameworks by up to 2.28%, while significantly achieving model volume reductions ranging from 20% to 80%. Our code is available at: https://github.com/jyzgh/DapperFL.

Main paper: 
 introduction: 
 Federated Learning (FL), an emerging distributed machine learning paradigm in edge computing environments [1, 2] , enables participant devices (i.e., clients) to optimize their local models while a central server aggregates these local models into a global model [3] . In contrast to traditional centralized machine learning paradigms, FL facilitates the collaborative training of a global model by distributed clients without the need for transmitting raw local data, thus mitigating privacy concerns associated with data transmission [2] . However, there are a number of challenges associated with FL deployments in edge computing settings, such as: a) System heterogeneity. Within prevalent FL environments, participant clients generally exhibit diverse and constrained system capabilities (e.g., CPU architecture, available memory, battery status, etc.). This heterogeneity often results in low-capability clients failing to complete local training of FL, consequently diminishing the performance of the aggregated global model [4, 5, 6] . b) Domain shifts. Owing to the distributed nature of FL, the data distributions among participant clients vary significantly [7, 8, 9] . Fig. 1 visually demonstrates the impact of these issues on FL performance, thus serving as the motivation for our work. Specifically, a) System heterogeneity: Device 1, with stringent resource constraints, fails to complete its model update within the maximum allowed time. As a result, the central server does not consider Device 1's model update, thus excluding it from the FL process and missing out on the data features captured by Device 1. b) Domain shifts: Devices 2 and 3 collect data from different domains, leading to diverse local models. With non-IID data, the global model aggregation may become biased, particularly if Device 3 contributes more data, thus potentially limiting the benefit for Device 2.Several studies have been devoted to addressing these challenges while only addressing each challenge independently. In addressing the system heterogeneity challenge, existing FL frameworks [10, 11, 12] compress local models for heterogeneous clients, thereby enabling the inclusion of low-capability clients into the FL process. However, these frameworks typically assume that the local data on clients shares a uniform domain, and the compressed models are tailored for each individual client. Such compressed models are susceptible to over-fitting on their local domains due to the presence of domain shifts in practice. Consequently, the central server struggles to aggregate a robust global model with strong Domain Generalization (DG) capabilities. To address the domain shifts challenge, existing solutions exemplified by works such as [13, 14, 15] , explore domain-invariant learning approaches either within the centralized machine learning paradigm or within ideal FL environments. However, these solutions unrealistically neglect the resource consumption of the clients, rendering them unsuitable for direct application in heterogeneous FL. Moreover, the inherent nature of FL yields non-independent and identically distributed (non-IID) data across the clients, further impeding these solutions from learning a domain-invariant global model, owing to limited data samples or labels available on individual clients. Despite existing work devoted to independently addressing system heterogeneity or domain shifts, few of them tackle these challenges simultaneously.To bridge the gaps observed in existing studies, we propose DapperFL, an innovative FL framework designed to enhance model performance across multiple domains within heterogeneous FL environments. DapperFL addresses the system heterogeneity challenge through the deployment of a dedicated Model Fusion Pruning (MFP) module. The MFP module generates personalized compact local models for clients by pruning them with fused knowledge derived from both the local domain and remaining domains, thus ensuring domain generalization of the local models. Additionally, we introduce a Domain Adaptive Regularization (DAR) module to improve the overall performance of DapperFL. The DAR module segments the pruned model into an encoder and a classifier, intending to encourage the encoder to learn domain-invariant representations. Moreover, we propose a novel model aggregation approach tailored to aggregating heterogeneous local models with varying architectures and weights. We summarize our contributions as follows:• We propose the MFP module to prune local models with personalized footprints leveraging both domain knowledge from local data and global knowledge from the server, thereby addressing the system heterogeneity issue in the presence of domain shifts. Additionally, we introduce a heterogeneous aggregation algorithm for aggregating the pruned models.• We propose the DAR module to enhance the performance of the pruned models produced by the MFP module. The DAR module introduces a regularization term on the local objective to encourage clients to learn robust representations across various domains, thereby adaptively alleviating the domain shifts problem.• We implement the proposed framework on a real-world FL platform and evaluate its performance on two benchmark datasets with multiple domains. The results demonstrate that Dap-perFL outperforms several state-of-the-art frameworks (i.e., [3, 16, 14, 15, 10, 17, 11, 12] ) by up to 2.28%, while achieving adaptive model volume reductions on heterogeneous clients.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The Abstract and Introduction sections accurately reflect the contribution and scope of the paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Heterogeneous Federated Learning. In heterogeneous FL, diverse system capabilities and data distributions across clients often result in performance degradation of the global model [18, 19, 20] .Extensive studies have made efforts to address these heterogeneity issues through various solutions.For example, studies in [10, 21, 22, 11, 23, 12] adopt model sparsification techniques to reduce the volume of local models, thereby involving low-capability clients in the FL process. The studies in [17, 24, 25] leverage dedicated objective functions and specialized training steps to address the data heterogeneity issue, with studies in [17, 25] allowing clients to conduct varying numbers of local updates and consequently alleviating system heterogeneity issue. Several studies [5, 26, 27] selectively optimize or transmit a fraction of the local model's parameters to reduce computational or communication resource consumption. Studies in [28, 29, 30] split the model into several sub-models and offload a subset of sub-models to the server for updating, therefore alleviating the training burden of clients. However, these heterogeneous FL frameworks commonly assume the data heterogeneity (i.e., non-IID data) exclusively involves distribution shifts in the number of samples and/or labels, while neglecting the existence of domain shifts.. DG is originally proposed in centralized machine learning paradigms to address the problem of domain shifts. Existing centralized studies assume access to the entire dataset during the model training process and propose various solutions to achieve DG [31] . For example, the studies in [32, 33, 34] focus on learning domain-invariant representations that can be generalized to unseen domains. In contrast to learning domain-invariant representations, several studies (e.g., [35, 36, 37, 38] ) train a generalizable model across multiple domains leveraging metalearning or transfer learning techniques. Additionally, there are studies (e.g., [39, 40, 13] ) that focus on the characteristics of domains, enhancing the generality of the model by properly augmenting the style of domain or instance. Unfortunately, the fundamental assumption of centralized machine learning is not satisfied in FL, where the dataset is distributed among clients who are restricted from sharing their data. Despite a few recent studies exploring DG approaches in FL [41, 14, 15, 9] through representation learning, prototype learning, and/or contrastive learning techniques, they typically neglect the inherent system heterogeneity feature of FL. Consequently, these approaches have shown limited effectiveness in heterogeneous FL due to strict resource constraints. In contrast, we explored a distributed model pruning approach that leverages both the local domain knowledge and the global knowledge from all clients, thereby reducing resource consumption in heterogeneous FL with the presence of domain shifts. Additionally, we design a specific regularization technique for updating the pruned local models, thereby further enhancing model performance across domains.3 DapperFL Design Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The discussion about the limitations of this work is provided in the Conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
model fusion pruning on edge devices: 
 In this subsection, we present the design of the MFP module employed by DapperFL. The goal of the MFP module is to tailor the footprint of the local model for edge devices in the presence of domain shifts in the local data, thereby addressing the system heterogeneity problem. Inspired by the spirit of the transfer learning [42, 43, 44] EQUATIONwhere α t is the fusion factor used to control the quality of the fused global model W t-1 . Considering that the local data on the client is typically limited, resulting to updating at the beginning of FL requires more guidance from the global model by exploring the commonalities across different domains. As the FL process proceeds, the local model is capable of learning more domain-dependent information from itself. Thus, we design a dynamic adjusting mechanism to modify the α t during the FL training. The α t is initialized with a relatively large value α 0 and decreases as the FL process proceeds. The decreased speed is controlled by a sensitivity factor ϵ. We assign a minimum value α min for α t to ensure the local model will consistently learn the knowledge from other domains. Formally, the dynamic adjusting mechanism for α t is described as follows:EQUATIONSubsequently, the MFP module calculates a binary mask matrix M t i ∈ {0, 1} |w t i | for pruning the fused local model w t i (in line 3). The matrix M t i is derived through the channel-wise ℓ 1 norm, which has proven to be effective and efficient in assessing the importance of parameters [47, 48] . The elements in M t i with a value of "0" indicate the corresponding parameters that need to be pruned, while those with a value of "1" indicate the parameters that will be retained. The pruning ratio ρ i determines the proportion of "0" in M t i . 3 Finally, the MFP module prunes the local model w t i with the binary mask matrix M t i (in line 4). The pruned model can be represented as w t i ⊙ M t i . It is noteworthy that, the pruning strategy used in our DapperFL is structural pruning strategy (channel pruning), which is a hardware-friendly approach that can be easily implemented with popular machine learning libraries such as PyTorch, making it particularly suitable for deployment on edge devices.In FL, each client i ∈ C possess private local data D i = {x i , y i } Ni , where x ∈ X denotes the input, y ∈ Y denotes the corresponding label, and N i represents the local data sample size. The data distribution p i (x, y) of client i typically varies from that of other clients, i.e., p i (x) ̸ = p j (x), p i (x|y) ̸ = p j (x|y), leading to the domain shifts problem. Due to the existence of domain shifts, representation z i generated by the local encoder varies among different clients, resulting in degraded prediction results of the local predictor. To address the domain shifts problem, we design a DAR module to enhance the performance of DapperFL across multiple domains while maintaining compatibility with the MFP module. Specifically, the DAR module introduces a regularization term to the local objective to alleviate the bias of representations z i on different clients adaptively. To achieve this goal, we first segment each pruned local model w⊙M into two parts, i.e., an encoder w e ⊙M e and a predictor w p ⊙M p , where w = {w e , w p } and M = {M e , M p }. 4 The encoder is responsible for learning a representation z i given an input x i , denoted as z i = g e (w e ⊙ M e ; x i ). While the predictor is responsible for predicting label ŷi given the representation z i , denoted as ŷi = g p (w p ⊙ M p ; z i ). Subsequently, we construct a regularization term L DAR on the local objective as follows:L DAR i = ||g e (w e ⊙ M e ; x i )|| 2 2 .(Here, || • || 2 2 represents the squared ℓ 2 norm of the local representation. The ℓ 2 norm implicitly encourages different local encoders to generate aligned robust representations adaptively, thereby mitigating the impact of domain shifts. We adopt the squared ℓ 2 norm to construct the regularization term for the following reasons: a) The ℓ 2 norm encourages each element of the representation to converge to 0 but not equal to 0. This property is advantageous compared to the ℓ 1 norm, which tends to make smaller representation elements exactly equal to 0. Thus, the squared ℓ 2 norm ensures that the pruned model retains more information in the representations. b) Higher order regularization introduces significant computational overhead compared to the ℓ 2 norm, without significantly improving the regularization effectiveness.Next, the cross-entropy loss used in the DAR module is constructed as:EQUATIONwhere K i denotes the set of possible labels on the client i, ŷi,k denotes predicting label, and y i,k denotes ground-truth label. Finally, the training objective of each client is calculated as follows:EQUATIONwhere the γ is a pre-defined coefficient controlling the importance of L DAR i relative to L CE i .Despite the MFP module and the DAR module being capable of alleviating domain shift issues, the heterogeneous local models generated by the MFP module cannot be aggregated directly using popular aggregation algorithms. Therefore, we propose a specific FL aggregation algorithm for DapperFL to effectively aggregate these heterogeneous local models.To preserve specific domain knowledge while transferring global knowledge to the local model, the central server first recovers the structure of local models before aggregating. Specifically, in each communication round t, the pruned local model is recovered as follows:EQUATIONwhere W t-1 is the global model aggregated at the (t -1)-th round, and M t i denotes the logical NOT operation applied to M t i . The first term w t i ⊙ M t i contains local knowledge5 specific to client i, while the second term w t-1 ⊙ M t i contains the global knowledge6 from all clients. Additionally, the structure of w t i ⊙ M t i , which includes local knowledge, complements w t i ⊙ M t i . Consequently, the structure recovery operation not only reinstates the pruned model's architecture but also transfers global knowledge to the pruned model, which is essential for the subsequent aggregation process. By combining these two forms of knowledge, we aim to leverage both the specialized insights of local models and the generalized capabilities of the global model, thereby enhancing the performance and adaptability of DapperFL. Finally, the global model is calculated by aggregating the recovered local models as follows:EQUATIONwhere |D i | is the sample number in the local dataset on client i, and |D| is the total number of samples in the entire FL system. Benefiting from the heterogeneous aggregation algorithm, the aggregated global model retains knowledge from all clients, enabling it to perform robustly across domains.The overall learning process of our proposed DapperFL is described in Algorithm 2.4 Experimental EvaluationQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: This paper describes the experimental environment in the Experimental Evaluation section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experimental setup: 
 Implementation Details. We implement DapperFL with a real-world FL platform FedML [49] with deep learning tool PyTorch [50] . We build the FL environment with a client set C containing 10 heterogeneous edge devices and a central server on the FedML platform. Following a convention setting [5, 25, 51] , we categorize these heterogeneous devices into 5 levels according to their system capabilities, i.e., C l ∈ C (l ∈ [1, 5] ), where C l represents device set belongs to level l. The system capabilities of devices in set C l decrease as level l increases. Our experiments are conducted on a GPU server with 2 NVIDIA RTX 3080Ti GPUs. Each experiment is executed three times with three fixed random seeds to calculate average metrics and ensure the reproducibility of our results.Datasets and Data Partition. We evaluate DapperFL on two domain generalization benchmarks, Digits [52, 53, 54, 55] and Office Caltech [56] that are commonly used in the literature for domain generalization. The Digits consists of the following four domains: MNIST, USPS, SVHN, and SYN. The Office Caltech consists of the following four domains: Caltech, Amazon, Webcam, and DSLR. For each benchmark, we distribute the simulated 10 clients randomly to each domain while guaranteeing that each domain contains at least one client and that each client only belongs to one domain. To generate non-IID local data, we randomly extract a proportion of data from the corresponding domain as the statistical characteristics vary among domains. Following the conventional data partition setting [15] , we set 1% as the local data proportion of Digits and 20% as that in Office Caltech based on the complexity and scale of the benchmarks.Models. We adopt ResNet10 and ResNet18 [57] as the backbone models for Digits and Office Caltech, respectively. In the following evaluations, both the DapperFL and comparison frameworks train the models from scratch for a fair comparison.Comparison Frameworks. We compare DapperFL with 8 state-of-the-art (SOTA) FL frameworks, including FedAvg [3] , MOON [16] , FedSR [14] , FPL [15] , FedDrop [10] , FedProx [17] , FedMP [11] , and NeFL [12] . These FL frameworks are either classical or focus on addressing system heterogeneity or domain shifts issues. The comparison frameworks are described in detail in Appendix A.Default Hyper-parameters. To conduct fair evaluations, the global settings and local training hyperparameters of FL are set to identical values to both the DapperFL and comparison FL frameworks.For the framework-specific hyper-parameters, we use their default settings without changing them.The hyper-parameters used in our evaluations are described in Appendix B.Evaluation Metrics. In this paper, we evaluate the performance of the global model using the average Top-1 accuracy across all domains. In evaluating the resource consumption of the clients, we adopt both the total number of parameters and the Floating-Point Operations (FLOPs) of the local model.Model Performance Across Domains. Tables 1 and 2 respectively provide detailed analyses of model accuracy on the Digits and Office Caltech benchmarks. The term "System Heter." indicates whether the respective framework supports system heterogeneity. 7 The results are reported as model accuracy with corresponding standard deviations in brackets.As shown in both Tables 1 and 2 , DapperFL achieves the highest global accuracy on both the Digits and Office Caltech benchmarks compared with the comparison frameworks. On Digits and Office Caltech, DapperFL's global accuracy is 0.13% and 2.28% better than the runner-up, respectively. This indicates that the global model learned by DapperFL is more robust across all domains and therefore possesses better domain generalization. Moreover, DapperFL always achieves competing accuracy on individual domains and even the best domain accuracy on the SYN, Amazon, and DSLR. This demonstrates that the DAR module of DapperFL implicitly encourages the encoder to learn domain-invariant representations, resulting in DapperFL being unlikely to over-fit on the specific domain(s) and possessing better domain generalization ability. Furthermore, DapperFL tolerates heterogeneous clients through personalized pruning ratio ρ, achieving adaptive resource consumption reductions of {20%, 40%, 60%, 80%} for clients in {C 2 , C 3 , C 4 , C 5 }, while outperforming comparison FL frameworks that support heterogeneous clients. This is evidence that the MFP module of DapperFL effectively reduces the resource consumption of clients with domain shifts. In addition, we provide learning curves for both global and domain accuracies in Appendix C for a comprehensive analysis. Impact of Model Footprints. In heterogeneous FL, resource constraints on the clients often result in limited model footprints (i.e., number of parameters and FLOPs). To investigate the impact of the model footprints on model performance, we compare the DapperFL with two SOTA FL frameworks (i.e., FedMP and NeFL) that are equipped with adaptive pruning techniques. The pruning ratio ρ determines both the reduction in the number of parameters and the FLOPs. We maintain a consistent ρ for every device, with predetermined values selected from the set {0.2, 0.4, 0.6, 0.8}.Figure 3 illustrates the comparative results under varying pruning ratios on both Digits and Office Caltech. As depicted, these frameworks tend to achieve higher accuracy with smaller pruning ratios, and DapperFL consistently outperforms the others across all ρ values on both benchmarks. This underscores the superior adaptability of DapperFL in heterogeneous FL environments. Notably, in Figure 3 (b), the accuracy of DapperFL at ρ = 0.4 surpasses its accuracy at ρ = 0.2. This discrepancy arises from the over-parameterization of ResNet18 for the classification task in Office Caltech. The improved accuracy observed at higher pruning rates suggests that the MFP module of DapperFL effectively prunes unnecessary parameters while enhancing model generalization rather than compromising it. This also demonstrates the reason behind DapperFL's superiority over comparison frameworks that employ full-size models. Furthermore, we comprehensively evaluate the effect of DapperFL's pruning ratio ρ on model performance in Appendix D. Effect of Hyper-Parameters within the MFP Module. We investigate the influence of the hyperparameters α 0 , α min , and ϵ within the MFP module on the overall performance of DapperFL. In this experiment, only the hyper-parameter under evaluation is modified, while others retain their default settings as outlined in Appendix B. Figure 4 (a) illustrates the effect of α 0 on model accuracy. The α 0 parameter dictates the initial weight of the global model fused into the local model. In both benchmarks, the optimal α 0 is evident at 0.9, resulting in the highest average accuracy. This result underscores the significance of infusing the local model with knowledge from other domains early in the FL process to enhance its generalization. Figure 4 (b) describes the effect of α min on model accuracy, where α min determines the minimum weight of the global model fused into the local model. In both benchmarks, the optimal average accuracy is achieved when α min is set to 0.1. This finding signifies that, as the FL process proceeds, the local model benefits from acquiring more domain-specific knowledge to uphold its personalization. Figure 4 (c) explores the effect of ϵ on model accuracy. The ϵ parameter controls the rate at which the fusion factor α 0 diminishes towards α min . In both benchmarks, the optimal ϵ is discernible at 0.2, resulting in the highest average accuracy. This observation emphasizes that the most effective rate of decrease for the fusion factor α 0 towards α min is achieved when ϵ = 0.2 in both benchmarks. Furthermore, recognizing the abnormal trends in the relationship between model accuracy and hyper-parameter ϵ when it is less than 0.2, we implement Bayesian search as an automatic selection mechanism to find the optimal value of ϵ. The results are provided in Appendix E.Effect of Hyper-Parameter within the DAR Module. Figure 4 (d) provides configurations with varying γ values across both Digits and Office Caltech. The results show an increase in accuracy with higher γ values until γ = 0.01 on Digits, where the highest accuracy of 74.30% is achieved. A notable drop in accuracy is observed at γ = 0.1, indicating that excessively large regularization may hinder model performance on Digits. Similar to Digits, an increase in accuracy is observed with higher γ values until γ = 0.01, reaching the highest accuracy of 67.75%. The impact of regularization is more pronounced on Office Caltech, as seen by the significant drop in accuracy at γ = 0.1. This suggests that the regularisation factor γ in the DAR Module plays a crucial role in solving the domain shift problem in FL and a careful choice of γ can improve the model performance. 
conclusion: 
 We have described our proposed FL framework DapperFL tailored for heterogeneous edge devices with the presence of domain shifts. Specifically, DapperFL integrates a dedicated MFP module and a DAR module to achieve domain generalization adaptively in heterogeneous FL. The MFP module addresses system heterogeneity challenges by shrinking the footprint of local models through personalized pruning decisions determined by both local and remaining domain knowledge. This effectively mitigates limitations associated with system heterogeneity. Meanwhile, the DAR module introduces a specialized regularization technique to implicitly encourage the pruned local models to learn robust local representations, thereby enhancing DapperFL's overall performance. Furthermore, we designed a specific aggregation algorithm for DapperFL to aggregate heterogeneous models produced by the MFP module, aiming to realize a robust global model across multiple domains.The evaluation of DapperFL's implementation on a real-world FL platform demonstrated that it outperforms several SOTA FL frameworks on two benchmark datasets comprising various domains.Limitations and Future Work. Despite its potential in addressing system heterogeneity and domain shifts, DapperFL introduces four hyper-parameters α 0 , α min , ϵ, and γ associated with the DG performance of the global model. A potential future direction involves the automatic selection of these hyper-parameters, therefore enhancing the flexibility and accessibility of the DapperFL. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The assets used in this paper are properly credited. The license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The comparison FL frameworks used in this work are introduced as follows:FedAvg [3] : FedAvg is a classical FL framework that is widely adopted to aggregate distributed local models. During the aggregation period of FedAvg, the clients upload the updates to the server, and the central server is responsible for generating the global model by performing weighted averaging on these local updates.FedDrop [10] : FedDrop reduces the computational overhead of local updating and communication costs by generating compact models for clients. In FedDrop, the server adopts lossy compression techniques to compress the global model with a uniform compression ratio for all clients. To satisfy the resource requirements of low-level clients, FedDrop needs to choose a large pruning ratio to compress the global model.FedProx FedSR [14] : FedSR incorporates two regularization techniques to implicitly align representations across domains in the FL environments. These regularizers implicitly align the marginal and conditional distributions of the representations, thereby enhancing domain generalization.NeFL [12] : NeFL employs a combination of depthwise and widthwise scaling techniques to partition a model into submodels. Its primary objective is to enable resource-constrained clients to seamlessly engage in the FL process, meanwhile facilitating the training of models on more extensive datasets. To mitigate the challenges associated with training multiple submodels featuring varying architectures, NeFL introduces a parameter decoupling mechanism. The default hyper-parameters used in our evaluations are described in Table 4 .We provide learning curves of global accuracy and domain accuracy in Figure 5 and Figure 6 , respectively. These curves provide the maximum Top-1 accuracy achieved until each communication round of FL. We systematically investigate the impact of varying pruning ratios (ρ) on the model performance of DapperFL using both the Digits and Office Caltech benchmarks. Pruning ratios ranging from 0.2 to 0.8 are considered, representing different degrees of model compression. Table 5 presents the model footprint and accuracy of DapperFL under different pruning ratios on the Digits benchmark.As the pruning ratio increases, the number of parameters (#Para) and FLOPs decrease, reflecting the expected reduction in model size and computational complexity. Notably, the model maintains competitive accuracy when pruning ratios increase from 0.2 to 0.6, indicating the robustness of DapperFL to varying degrees of model compression. Similarly, Table 6 details the model footprint and accuracy of DapperFL under different pruning ratios on the Office Caltech benchmark. Consistent with the Digits benchmark results, the model adapts effectively to increased pruning from 0.2 to 0.6, with reductions in both parameters and FLOPs.In summary, DapperFL consistently outperforms the baseline frameworks that utilize the entire model, such as FedAvg and MOON, as illustrated in Table 1 and Table 2 , when employing a pruned model with a pruning ratio ρ ranging from 0.2 to 0.6. Despite incurring an inevitable accuracy loss at ρ = 0.8 on both benchmarks, DapperFL still outperforms FedMP and NeFL, both equipped with adaptive pruning capabilities, as illustrated in Figure 3 . These observations collectively demonstrate the robustness of DapperFL across a spectrum of pruning ratios, emphasizing its potential for deployment in resource-constrained FL environments. The ability to achieve substantial model compression (with the pruning ratio ρ ranging from 0.2 to 0.6) without significant loss in accuracy makes DapperFL a promising solution for real-world applications where resource efficiency is paramount. We run DapperFL on the Office Caltech benchmark 40 times, adopting a distinct ϵ of less than 0.2 each time. The values are selected using the Bayesian search. The results are presented in Fig. 7 . As illustrated, the Bayesian search-based automatic selection mechanism indicates that model accuracy is likely to reach a higher level when ϵ approaches 0.2, aligning with our default setting of ϵ = 0.2. It is noteworthy that, owing to the nature of Bayesian search, the sampled ϵ values are not uniformly distributed, as they tend to bias towards the optimal ϵ that maximizes model accuracy.ϵNeurIPS Paper ChecklistJustification: This paper conforms to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).In the first communication round, the initial backbone model is used as the global model.Following the conventional setting in heterogeneous FL[21,11,5,12], we make the fundamental assumption that the system capabilities of the devices are available to the server and the pruning ratios for all devices are appropriately determined according to the system information.We omit the client index i and the communication round index t for notation simplicity. In this work, all layers except the final linear layer act as the encoder, while the last linear layer of the model acts as the predictor.In this work, "local knowledge" refers to the feature extraction capabilities of the local model, which are learned from the specific data available in its local domain. This knowledge encapsulates the nuances and characteristics of the data that the local model has been trained on.The "global knowledge" represents the aggregated feature extraction capabilities of the global model, which are informed by data across all participating domains. The global model synthesizes diverse knowledge from different local models to provide a more generalized understanding that is applicable across multiple domains.Note that when evaluating frameworks that do not support heterogeneous clients, we allow all clients to participate in FL by disregarding resource constraints on clients. 


Decision: The paper proposes a novel application scenario. The paper is well written. The unclear parts are resolved by rebuttal. All reviewers agreed to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel federated learning framework to address system heterogeneity and domain shifts in edge computing environments. The framework employs a Model Fusion Pruning (MFP) module to generate personalized compact local models and a Domain Adaptive Regularization (DAR) module to enhance performance across multiple domains. Experimental results demonstrate the effectiveness of the proposed method in terms of accuracy improvements and model compression.
strength_weakness: strength_weakness: 1. The proposed method is sensitive to hyperparameter selection. For example, the framework introduces several hyper-parameters (α0, αmin, ϵ, γ) that significantly influence performance, which also complicates the application of DapperFL. As pointed out in the “Limitations” of this manuscript, it could benefit from providing a set of default values for these hyper-parameters or an automatic hyper-parameter selection mechanism. 

2. Experimental Results Analysis: Despite DapperFL achieving competitive model accuracy with a significantly smaller model footprint, it would be beneficial to discuss why a smaller model could outperform a larger one. 

3. It is unclear how the proposed meethod to be applied to new devices in test or production phase.

4. Missing some related work, such as heterogenous FL [1,2] and model pruning on devices [3,4].

[1] Zhuangdi Zhu, et al., Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning (pp. 12878-12889). ICML 2021.

[2] Yue Tan, et al., Federated Prototype Learning across Heterogeneous Clients, AAAI 2022

[3] Yuang Jiang, et al., Model Pruning Enables Efficient Federated Learning on Edge Devices, IEEE TNNLS 2022

[4] Haiyan Zhao, et al., One-Shot Pruning for Fast-adapting Pre-trained Models on Devices, arXiv preprint arXiv:2307.04365
questions: questions: 1. Any suggestions when choosing hyperparameters (α0, αmin, ϵ, γ) for good global model performance in DapperFL? 

2. Why does the smaller model generated by DapperFL have an accuracy comparable to that of SOTA, and sometimes even exceed the accuracy of a full-size model? 

3.  In eq.6, the authors combine “local knowledge” and “global knowledge” to recover the local model. It could be better to introduce these definitions more clearly before using them. What do “local knowledge” and “global knowledge” in DapperFL stand for, and where are they derived from?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes DapperFL, an innovative FL framework designed to enhance model performance across multiple domains within heterogeneous FL environments. DapperFL addresses the system heterogeneity challenge through the deployment of a dedicated Model Fusion Pruning (MFP) module. Additionally, a Domain Adaptive Regularization (DAR) module is introduced to improve the overall performance of DapperFL. The evaluation results are positive compared with SOTA.
strength_weakness: strength_weakness: W1: It could be better to provide a discussion on why only one epoch is enough to fine-tune the global model in Algorithm 1.
W2:	In the evaluation section, the authors should provide more framework-specific default values for hyper-parameters.
W3: The subscript “i” is used to denote both client subscripts in the methods section and client-level subscripts in the experiments section. It would be better to use different notation.
W4: The limitations of this work could be elaborated more to let the audience develop a comprehensive understanding.
questions: questions: Q1: Why is the l_1 norm (rather than other normalization methods such as the l_2 norm) used to calculate mask matrices?
Q2: When DapperFL aggregates heterogenous models on the central server, how does the sever know the architecture of these heterogenous models?
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The authors propose a new FL scenario that faces challenges of both system heterogeneity and domain shifts. This situation means that the clients have varying capabilities and their data domains also differ. The authors claim that they propose three novel modules to tackle the challenges: Model Fusion Pruning (MFP), Domain Adaptive Regularization (DAR), and a specific aggregation algorithm. The MFP module aims to address the challenge of system heterogeneity. It first fine-tunes the global model with local data to get a local model and then fuses the local model and the global model using a simple weighted average, where the fusion factor decays with the training epochs. Then, it prunes the fused model based on the L1 norm and continues training this model on the local data. The DAR module aims to address the challenge of domain shifts. It adds an L2 regularization on the first few layers of the fused model during the training process. To aggregate the locally pruned models, the models' structure will be discovered using the weights of the global model (of the last communication round) before global aggregation. The authors evaluate their method on two domain generation datasets, and the experimental results show that DapperFL achieves a SOTA performance.
strength_weakness: strength_weakness: 1. This paper lacks significant innovation. The proposed framework is just a combination of existing approaches to solving the problems of domain shifts and system heterogeneity, respectively. It’s common to use model pruning to solve the problem of system heterogeneity, such as [1][2][3]. The model aggregation method seems natural for updating the subnetwork of the local model on the global model and has already been proposed in similar work such as [3].
2. The explanation of the proposed method is unclear. For example, the Domain Adaptive Regularization module does not specify how the model is segmented.
3. The motivation of this paper is not strong enough. This paper lacks evidence to demonstrate the importance of the scenarios present in the Introduction section (with both system heterogeneity and domain heterogeneity).

[1] Jiang, Z., Xu, Y., Xu, H., Wang, Z., Liu, J., Chen, Q., & Qiao, C. (2023). Computation and communication efficient federated learning with adaptive model pruning. *IEEE Transactions on Mobile Computing*, *23*(3), 2003-2021.
[2] Li, A., Sun, J., Zeng, X., Zhang, M., Li, H., & Chen, Y. (2021, November). Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. In *Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems* (pp. 42-55).
[3] Li, A., Sun, J., Li, P., Pu, Y., Li, H., & Chen, Y. (2021, October). Hermes: an efficient federated learning framework for heterogeneous mobile clients. In *Proceedings of the 27th Annual International Conference on Mobile Computing and Networking* (pp. 420-437).
questions: questions: 1. How do you segment the model in the Domain Adaptive Regularization module?
2. Do you have any evidence showing the importance or prevalence of the scenario present in the Introduction section (FL with both system and domain heterogeneity)?
limitations: limitations: The authors discuss the limitation that the proposed method introduces four hyper-parameters and the choice of these hyperparameters significantly impacts the performance of the method.
paper_summary: paper_summary: This paper proposes DapperFL, a novel framework for domain adaptive federated learning designed specifically for heterogeneous edge devices. It addresses the challenge of system heterogeneity and domain shifts in FL by integrating a Model Fusion Pruning (MFP) module and a Domain Adaptive Regularization (DAR) module. The framework also incorporates a novel aggregation approach to handle heterogeneous local models with varying architectures and weights.
strength_weakness: strength_weakness: •	The introduction of the two issues, system heterogeneity and domain shift, in this article are not very clear. Regarding system heterogeneity, the author places emphasis on low-capability clients being unable to perform local training, but this emphasis lacks relevance to the rest of the paper. Additionally, the example of the cameras in the domain shifts problem is somewhat biased towards the "multi-view task." For the issues of model heterogeneity and domain shift, the author needs to provide a clearer exposition of the application scenarios where these two problems intersect. If feasible, it would be beneficial to supplement the explanation with illustrations.

•	The strategy employed in the Model Fusion Pruning phase is not particularly innovative. Moreover, this strategy requires the storage of the entire global model and additional extensive computations under resource-constrained settings, which severely contradicts the problem that this paper aims to address. Furthermore, the pruning strategy employed in this article is the unstructured pruning strategy, so the actual contribution to the problem of system heterogeneity is limited.

•	Althought the framework diagram is already very clear, it is suggested to include model symbols corresponding to the algorithm in order to provide a clearer explanation of the model's changes at each step.

•	The method details in this article bear some resemblance to existing work, and there is a lack of references to these works. ( Diao, Enmao, Jie Ding, and Vahid Tarokh. "Heterofl: Computation and communication efficient federated learning for heterogeneous clients." arXiv preprint arXiv:2010.01264 (2020), Yuang Jiang, Shiqiang Wang, Bong Jun Ko, Wei-Han Lee, and Leandros Tassiulas. 2019. Model Pruning Enables Efficient Federated Learning on Edge Devices. CoRR abs/1909.12326 (2019). )

•	In addition, there are some approaches that address model heterogeneity and domain shift problems through distillation. Although this paper focuses on pruning paths, these methods should also be mentioned and even compared. (Zhu, Zhuangdi, Junyuan Hong, and Jiayu Zhou. "Data-free knowledge distillation for heterogeneous federated learning." International conference on machine learning. PMLR, 2021. Huang, Wenke, Mang Ye, and Bo Du. "Learn from others and be yourself in heterogeneous federated learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.)
questions: questions: •	Is there a practical difference between the fine-tuning global model steps inspired by the spirit of transfer learning in this article and the local model training after initializing the global model in regular federated learning? Can you provide a more specific explanation of the role of transfer learning in this step?

•	The global model deployment step appears to result in the global model being stored on the local client and requiring fine-tuning as well as mask operations for traversing each neural unit. Does this contradict resource-constrained scenarios in heterogeneous systems?
limitations: limitations: •	The authors pointed out the limitations of the paper in terms of hyper-parameters selection. However, this article still has certain limitations in its solution for system heterogeneity. In addition, it is necessary to add theoretical conversion analysis of the proposed federated learning algorithm.

Comments: title: Response to the rebuttal
comment: Thanks for the detailed responses. I'm generally fine with the results.  Thus, I will keep my score.
Thanks for your detailed rebuttal response. In the rebuttal, the authors have elaborated on the novelty of this paper, and I think their claims are reasonable. Besides, the authors have clarified the previously unclear description of the methodology, i.e. the model's segmentation approach. Finally, the authors have provided specific and meaningful examples, demonstrating that the scenarios in the paper are common and significant. Therefore, our issues have been well addressed and we are satisfied with the rebuttal response. We will raise our minor scores to 3/3/3 points and the overall rating to 6 (Weak Accept).
Thanks for the authors’ comprehensive rebuttal and the detailed illustrations provided in the pdf file. 

The responses have effectively addressed my concerns. The clarification on system heterogeneity was particularly helpful. The relevance of DapperFL to the system heterogeneity concern is solved by distinguishing between low-capability clients "failing to complete" training rather than being "unable to perform" it and explaining how MFP mitigates this issue. Additionally, the emphasis on domain shifts and the examples provided have improved the clarity of the scenarios where these issues intersect. My concerns about the suitability of DapperFL is solved by the authors’ clarification on the channel pruning strategy. The authors have mentioned that, DapperFL substantially reduces computation costs in subsequent epochs aside from the minimal and necessary initial costs, which is a significant contribution to the target scenarios. The inclusion of additional references to related work, as well as the reasoning for not focusing on distillation methods, was also satisfactory. Also, I appreciate the decision to include relevant references in the final version. I hope the materials in the rebuttal can be integrated into the final version of the paper to facilitate interested readers’ understanding of the framework.

Given the thoroughness and clarity of the responses, I am satisfied that my concerns have been adequately addressed and happy to lift my evaluation of this submission.