DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices
Authors: Yongzhe Jia
Xuyun Zhang
Hongsheng Hu
Kim-Kwang Raymond Choo
Lianyong Qi
Xiaolong Xu
Amin Beheshti
Wanchun Dou
Venue: NeurIPS 2024 oral (2024)
Keywords: Federated learning
Model pruning
Domain adaptation
Edge intelligence
Primary area: infrastructure
TL;DR: A heterogeneous FL framework DapperFL to enhance model performance across multiple domains on resource-limited edge devices.

Abstract: Federated learning (FL) has emerged as a prominent machine learning paradigm in edge computing environments, enabling edge devices to collaboratively optimize a global model without sharing their private data. However, existing FL frameworks suffer from efficacy deterioration due to the system heterogeneity inherent in edge computing, especially in the presence of domain shifts across local data. 
In this paper, we propose a heterogeneous FL framework DapperFL, to enhance model performance across multiple domains. In DapperFL, we introduce a dedicated Model Fusion Pruning (MFP) module to produce personalized compact local models for clients to address the system heterogeneity challenges. The MFP module prunes local models with fused knowledge obtained from both local and remaining domains, ensuring robustness to domain shifts. Additionally, we design a Domain Adaptive Regularization (DAR) module to further improve the overall performance of DapperFL. The DAR module employs regularization generated by the pruned model, aiming to learn robust representations across domains. Furthermore, we introduce a specific aggregation algorithm for aggregating heterogeneous local models with tailored architectures and weights. We implement DapperFL on a real-world FL platform with heterogeneous clients. Experimental results on benchmark datasets with multiple domains demonstrate that DapperFL outperforms several state-of-the-art FL frameworks by up to 2.28%, while significantly achieving model volume reductions ranging from 20% to 80%. Our code is available at: https://github.com/jyzgh/DapperFL.

DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices
Yongzhe Jia', Xuyun Zhang', Hongsheng Hu', Kim-Kwang Raymond Choo',
Lianyong Qi Xiaolong Xu Amin Beheshti', Wanchun Dou'
State Key Laboratory for Novel Software Technology,
Department of Computer Science and Technology, Nanjing University, China
2 School of Computing, Macquarie University, Australia School of Information and Physical Sciences, University of Newcastle, Australia
Department of Information Systems and Cyber Security, University Texas at San Antonio, USA
College of Computer Science and Technology, China University Petroleum (Fast China), China School of Computer and Software, Nanjing University of Information Science and Technology, China
Abstract
Federated learning (FL.) has emerged as prominent machine learning paradigm in edge computing environments, enabling edge devices to collaboratively optimize a global model without sharing their private data. However, existing FL. frameworks suffer from efficacy deterioration due to the system heterogeneity inherent in edge computing, especially in the presence of domain shifts across local data. In this paper, we propose heterogeneous FL framework DapperFL, to enhance model performance across multiple domains. In DapperFL, introduce dedicated Model Fusion Pruning (MFP) module to produce personalized compact local models for clients to address the system heterogeneity challenges. The MFP module prunes local models with fused knowledge obtained from both local and remaining domains, ensuring robustness domain shifts. Additionally, design Domain Adaptive Regularization (DAR) module to further improve the overall performance of DapperFI.. The DAR module employs regularization generated by the pruned model, aiming to learn robust representations across domains. Furthermore, we introduce a specific aggregation algorithm for aggregating heterogeneous local models with tailored architectures and weights. We implement DapperFL. on a real- world FL platform with heterogeneous clients. Experimental results on benchmark datasets with multiple domains demonstrate that DapperFL outperforms several state-of-the-an FL frameworks by up to 2.28%. while significantly achieving model volume reductions ranging from 20% to 80%. Our code is available at: https: tig........... cww Saam/jyygg//DapprFf
Introduction
Federated Learning (FL), an emerging distributed machine learning paradigm in edge computing environments 2) enables participant devices (i.e., clients) optimize their local models while a central server aggregates these local models into global model 3 In contrast to traditional centralized machine learning paradigms, FL. facilitates the collaborative training of global model by distributed clients without the need for transmitting raw local data, thus mitigating privacy concerns associated with data transmission (2)
"Corresponding author: Xiaolong Xu (xixu@mmist.cdu.cc)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Accuracy on domain 2
Accuracy on domsin 3
Aggregated
Global model
Central Server
global model
performance degradation
Global model
Global model
Local model
Global model
Local
model
Local update
Local update on Device
Local update on Device 3
on Device
Local data for domuin
Local data for domain 3
Update timeout
Figure A motivational example of DapperFL. with heterogeneous devices and multiple domains.
However, there are a number of challenges associated with FL deployments edge computing settings, such as: a) System heterogeneity. Within prevalent FL environments, participant clients generally exhibit diverse and constrained system capabilities (c.g., CPU architecture, available memory, battery status, etc.). This heterogeneity often results in low-capability clients failing to complete local training of FL, consequently diminishing the performance of the aggregated global model 4 5 6. b) Domain shifts. Owing to the distributed nature of FL, the data distributions among participant clients vary significantly 7 9.. Fig. T visually demonstrates the impact of these issues on FL performance, thus serving as the motivation for our work. Specifically, a) System heterogeneity: Device I, with stringent resource constraints, fails to complete its model update within the maximum allowed time. As result, the central server does not consider Device I's model update, thus excluding it from the FL process and missing out on the data features captured by Device 1. b) Domain shifts: Devices 2 and 3 collect data from different domains, leading diverse local models. With non-III data, the global model aggregation may become biased, particularly if Device 3 contributes more data, thus potentially limiting the benefit for Device
Several studies have been devoted to addressing these challenges while only addressing each challenge independently. In addressing the system heterogeneity challenge, existing FL frameworks 10 compress local models for heterogeneous clients, thereby enabling the inclusion low-capability clients into the FL process. However, these frameworks typically assume that the local data clients shares a uniform domain, and the compressed models are tailored for each individual client. Such compressed models are susceptible over-fitting on their local domains due the presence of domain shifts in practice. Consequently, the central server struggles to aggregate robust global model with strong Domain Generalization (DG) capabilities. To address the domain shifts challenge. existing solutions exemplified by works such as [13] [4, [5], explore approaches either within the centralized machine learning paradigm within ideal FL environments. However, these solutions unrealistically neglect the resource consumption of the clients, rendering them unsuitable for direct application heterogeneous FL. Moreover, the inherent nature of FL yields non con-independent and identically distributed (non-III -ID) data across the clients, further impeding these solutions from learning a labels available on individual clients. Despite existing work devoted to independently addressing system heterogeneity or domain shifts, few of them tackle these challenges simultaneously.
[1 12
learning
global model, owing to limited data samples or
To bridge the gaps observed in existing studies, we propose DapperFI., an innovative FL framework designed enhance model performance across multiple domains within heterogeneous FL envi- ronments. DapperFL addresses the system heterogeneity challenge through the deployment of dedicated Model Fusion Pruning (MFP) module. The MFP module generates personalized compact local models for clients by pruning them with fused knowledge derived from both the local domain and remaining domains, thus ensuring domain generalization of the local models. Additionally, we introduce a Domain Adaptive Regularization (DAR) module to improve the overall performance of DL. The DAR module segments the pruned model into an encoder and a classifier, intend- ing to encourage the encoder to learn domain-invarian representations. Moreover, we propose novel model aggregation approach tailored to aggregating heterogeneous local models with varying architectures and weights. We summarize our contributions as follows:

---

We propose the MFP module to prune local models with personalized fooiprints leveraging both domain knowledge from local data and global knowledge from the server, thereby addressing the system heterogeneity issue in the presence of domain shifts. Additionally, we introduce a heterogeneous aggregation algorithm for aggregating the pruned models.
We propose the DAR module to enhance the performance of the pruned models produced by the MFP module. The DAR module introduces regularization term on the local objective to encourage clients to learn robust representations across various domains, thereby adaptively alleviating the domain shifts problem.
We implement the proposed framework on a real-world FL platform and evaluate its perfor- mance on two benchmark datasets with multiple domains. The results demonstrate that Dap- perFL outperforms several state-of-the-an frameworks (i.c., [3] 16 [4 by up to 2.28%, while achieving adaptive model volume reductions on heterogencous clients.
15 10 17 12)
2 Related Work
Heterogeneous Federated Learning. In heterogeneous FL, diverse system capabilities and data distributions across clients often result in performance degradation of the global model [8] 19 20 Extensive studies have made efforts to address these heterogeneity issues through various solutions.
For example, studies in [O 21 22 23 [2] adopt model sparsification techniques to reduce the
volume of local models, thereby involving low-capability clients in the FL process. The studies in 17, 24 225 leverage dedicated objective functions and specialized training steps to address the
data heterogeneity issue, with studies in [17 (5 allowing clients to conduct varying numbers of
local updates and consequently alleviating system heterogeneity issue. Several studies [5] 26 (2) selectively optimize or transmit fraction of the local model's parameters reduce computational or
split the model into several sub-models and offload a subset of sub-models to the server for updating. therefore alleviating the training burden of clients. However, these heterogeneous FL frameworks commonly assume the data heterogeneity (i.e.. non-IlI data) exclusively involves distribution shifts in the number of samples and/or labels, while neglecting the existence of domain shifts.
communication resource consumption. Studies in [28 29 30
Domain Generalization (DG). DG is originally proposed in centralized machine learning paradigms to address the problem of domain shifts. Existing centralized studies assume access to the entire dataset during the model training process and propose various solutions achieve DG [I]]. For example, the studies in [32
focus on learning domain-invariant representations that can be generalized to unseen domains. In contrast to learning domain-invarian representations, several studies (e.g., [35 36 37 (8) train a generalizable model across multiple domains leveraging meta- learning or transfer learning techniques. Additionally, there are studies (e.g.. [39 40 [3]
33 34
that focus on the characteristics of domains. enhancing the generality of the model by properly augmenting the style of domain or instance. Unfortunately. the fundamental assumption of centralized machine learning is not satisfied in FL., where the dataset is distributed among clients who are restricted from sharing their data. Despite a few recent studies exploring DG approaches in FL representation learning, prototype learning, and/or contrastive learning techniques, they typically neglect the inherent system heterogeneity feature of FL. Consequently, these approaches have shown limited effectiveness in heterogeneous FL due to strict resource constraints. In contrast, we explored a distributed model pruning approach that leverages both the local domain knowledge and the global knowledge from all clients, thereby reducing resource consumption in heterogeneous FI. with the presence of domain shifts, Additionally, we design a specific regularization technique for updating the pruned local models, thereby further enhancing model performance across domains.
15 9 through
3 DapperFL Design
3.1 Overview of DapperFL
DapperFL comprises two key modules, namely Model Fusion Pruning (MFP) and Domain Adaptive Regularization (DAR). The MFP module is designed to compress the local model while concurrently addressing domain shift problems, and the DAR module employs regularization generated by the compressed model to further mitigate domain shift problems and hence improve the overall perfor-
3

---

Model Puston Prusing
Update local model with Domain Adaptive Rogulatization
Model Pasion Pruning
Update local model with Domnin Adeptive Regularization
Client
Local objective C
Clicet
Local objective
Dousai--peeifie representations
Domaain-qpeefie represertatiions
Fused noem pouning!
praniing
Imput
Input
Fine-luned local model
Fine tuned
Pruned model
Local data
Encoder Classifeer
local model
Pruned model
Local data
Racoder
Clessidee
Global model
Updated local model
Global model
Updated local model
Local model on elicat
model on clical
B
Recover
Aggregate local models
model structure
Aggregated global model for subsoqueett round
Contral
Local model cliene
Local model
client
Figure 2: Overview of DapperFI. framework with two clients for each communication round.
mance of DapperFL. Additionally, to handle the aggregation of heterogeneous models produced by the MFP module, we introduce a dedicated heterogeneous model aggregation algorithm.
Figure 2 explains the DapperFL's workflow during each communication round which is also described as follows. ① The MFP module within each client i E C calculates the fusion model w using both the global and fine-tuned local models. The global model W is downloaded from the central server and the local model wo is fine-tuned the client's local data with initial epoch. Subsequently, the MFP calculates a binary mask matrix M using Ê norm and produces the pruned local model w @ M @ The DAR module updates the pruned model w C M over several epochs with a dedicated local objective C. During local updating, the DAR module segments the local model into an encoder and a classifier, which are responsible for generating local representations and performing predictions, respectively. The local objective C, is then constructed by combing regularization C.AR of the local representations with the common cross-entropy loss ECE. Next, the client transmits the updated local model to the central server. @ To aggregate local models with heterogeneous structures, the central server recovers the structure of received local models using the previous round's global model w' 4 The central server aggregates the recovered local models through weighted averaging, obtaining the global model W for round l.
3.2 Model Fusion Pruning on Edge Devices
In this subsection, we present the design the MFP module employed by DapperFL. The goal of the MFP module is to tailor the footprint of the local model for edge devices in the presence of domain shifts in the local data, thereby addressing the system heterogeneity problem. Inspired by the spirit of the transfer learning 42
MFP fuses the global model W into the fine-tuned local model w to learn the cross-domain knowledge. This avoids over-fitting the model to the local domain while enhancing the generality of the model. After that, MFP calculates a binary mask matrix M to generate the pruned local model. The detailed pruning process described in Algorithm [
43
44]
Specifically, in the initial epoch of each communication round E [1. the MFP module first fine-tunes the global model W on local data D; to produce local model w (in line 1). We utilize epoch of local training to determine the pruned models for the following reasons: Additional local epochs do not significantly enhance the model's 's performance, which justifies the use of single epoch for efficiency. As noted [45), experiments have demonstrated that extending local training beyond one epoch yields results comparable to those achieved with just one epoch. 2) In previous domain generalization-reltte domain information. This method has proven adequate for capturing essential features and domain characteristics. 3) Pioneering research in model design and neural architecture search, such as [46]
FL research, such as 45). one epoch is also employed to collect local
In the first communication round, the initial backbone model is used the global model.

---

Algorithm 1 Model Fusion Pruning of DapperFL
Input: Global model W local data D,, pruning ratio Ps
Output: Pruned local model w @ M
I: o +- Fine-tune global model w on local data D;
2: w f- Fuse the global model W into the local model wo using Eq. and Eq.
3: M Calculate binary mask matrix by & norm with pruning ratio p;
4: wi @ M +-
Prune the local model w with binary mask matrix M
5: return Pruned local model w O M
has demonstrated that few epochs are sufficient to obtain coarse estimate of sub-model. This approach is effective in quickly assessing model configurations without extensive computation.
Next, the MFP module fuses the global model W into the local model u embed the cross- domain information into the local model (in line 2). The fusion operation is formulated as follows:
: -1-
+
(1-a')o
where a' is the fusion factor used to control the quality of the fused global model W Considering that the local data on the client is typically limited, resulting to updating at the beginning of FL requires more guidance from the global model by exploring the commonalities across different domains. As the FL process proceeds, the local model is capable of learning more domain- information from itself. Thus, we design a dynamic adjusting mechanism to modify the a during the FL training. The a² is initialized with a relatively large value ag and decreases as the FL process proceeds. The decreased speed is controlled by a sensitivity factor e. We assign a minimum value Omin for a' to ensure the local model will consistently learn the knowledge from other domains. Formally, the dynamic adjusting mechanism for a' is described as follows:
a = max((1 (1- (). ag. Ommin
(2)
Subsequently, the MFP module calculates binary mask matrix M; E (0, 13 for pruning the fused local model wi (in line 3). The matrix M; is derived through the channel-wise ε norm, which has proven to be effective and efficient in assessing the importance of parameters (47 48). The elements in M with value of "0" indicate the corresponding parameters that need to be prumed, while those with a value of "1" indicate the parameters that will be retained. The pruning ratio p; determines the proportion of "0" in M' F Finally, the MFP module prunes the local model w with the binary mask matrix M (in line 4). The pruned model can be represented w @ M It is noteworthy that, the pruning strategy used in our DapperFL. is structural pruning strategy (channel pruning), which is hardware-frienm learning libraries such as Py Torchh, making it particularly suitable for deployment on edge devices.
approach that can be easily implemented with popular machine
3.3 Domain Adaptive Regularization
In FL, each client i E C possess private local data D; = {x N: where x E X denotes the input, y E y denotes the corresponding label. and N, represents the local data sample size. The data distribution (x of client typically varies from that of other clients, i.e., pa(ar) * py (x), (x(y))) # (*ly),, leading the domain shifts problem. Due to the existence of domain shifts, representation 2; generated by the local encoder varies among different clients, resulting in degraded prediction results of the local predictor. To address the domain shifts problem, we design a DAR module to enhance the performance of DapperFL across multiple domains while maintaining compatibility with the MFP module.
Specifically, the DAR module introduces a regularization term to the local objective to alleviate the bias of representations on different clients adaptively. To achieve this goal, we first segment each pruned local model w O M into two parts, i.c., an encoder we M. and a predictor w. O Mp. where
'Following the conventional setting in heterogeneous FL 21 12 we make the fundamental assumption that the system capabilities of the devices are available to the server and the pruning ratios for all devices are appropriately determined according to the system information.
5

---

w = (we, wp) and M = (Mc. Mp) p The encoder is responsible for learning a representation 2, given an imput 2,, denoted as 2; 9r (w, a; zz). While the predictor responsible for denoted as = 9p 9p(Wpp & M pi 2.). Subsequently, we construct regularization term DAR on the local objective as follows:
predicting label Vi given the representation
Cp
ge (we C M. Me:z))ll.
Here, 11 represents the squared & norm of the local representation. The & norm implicitly encourages different local encoders to generate aligned robust representations adaptively, thereby mitigating the impact of domain shifts. We adopt the squared lg norm construct the regularization term for the following reasons: a) The l2 norm encourages each element of the representation to converge 0 but not equal to O. This property is advantageous compared to the & norm, which tends to make smaller representation elements exactly equal to Thus, the squared lz norm ensures that the pruned model retains more information in the representations. b) Higher order regularization introduces significant computational overhead compared to the & norm, without significantly improving the regularization effectiveness.
Next, the cross-entropy loss used in the DAR module is constructed as:
C
Σ kEK,
Vo,4 log (og((),k);
(4)
KGG
where K;; denotes the set of possible labels on the client i, Ii,k denotes predicting label, and yı.k denotes ground-truth label. Finally, the training objective of each client is calculated as follows:
= +AR,
(5)
where the y a pre-defined coefficient controlling the importance of CPAR relative LCE.
3.4 Heterogeneous Model Aggregation
Despite the MFP module and the DAR module being capable of alleviating domain shift issues, the heterogeneous local models generated by the MFP module cannot be aggregated directly using popular aggregation algorithms. Therefore, we propose specific FL aggregation algorithm for DapperFL. effectively aggregate these heterogeneous local models.
To preserve specific domain knowledge while transferring global knowledge to the local model, the central server first recovers the structure of local models before aggregating. Specifically, in each communication round f, the pruned local model is recovered as follows:
w = wi M + W о M
local knowledge
global knowledge
where W the global model aggregated at the (t (t- 1)-th round, and M denotes the logical NOT
operation applied to M The first term wi @ M contains local ShowledgeFsspecinn specific to client i, while the second term ro-11o C M contains the global knowledge from all clients. Additionally, the structure of wi @ M which includes local knowledge, complements w ( M Consequently, the structure recovery operation not only reinstates the pruned model's s architecture but also transfers global knowledge to the pruned model, which is essential for the subsequent aggregation process. By combining these two forms of knowledge, we aim to leverage both the specialized insights of local models and the generalized capabilities of the global model, thereby enhancing the performance and adaptability of DapperFL.
"We omit the client index and the communication round index for notation simplicity. In this work, all layers except the final linear layer act the encoder, while the last linear layer of the model acts the predictor.
'In this work, "local knowledge" refers to the feature extraction capabilities of the local model, which are learned from the specific data available in its local domain. This knowledge encapsulates the nuances and characteristics of the data that the local model has been trained
'The "global knowledge" represents the aggregated feature extraction capabilities of the global model, which are informed by data across all participating domains. The global model synthesizes diverse knowledge from different local models provide more generalized understanding that is applicable across multiple domains.

---

Algorithm 2 DapperFL
Input:
Initial global model W total number of communication rounds T, total number of local
epochs E, local dataset D; and pruning ratio p; for each client
Output: Optimized global model W
Local Updates:
1: wi ( M + Prune local model by the pruning ratio pr using Algorithm local epoch e =
2: for local epoch e E [2, F.] do
3: Update the pruned local model w M using local dataset D; and local objective Eq
4: end for
Server Executes:
5: for communication round E [1, do
6: Send global model W to all clients // waiting for the clients finish updating
7: Receive updated models w @ M; from all clients
8: un Recover the local models using Eq
9: W <-
Aggregate the local models using Eq
end for
11: return Optimized global model W
Finally, the global model is calculated by aggregating the recovered local models as follows:
D D
W
W
IEC
where [D.] is the sample number in the local dataset on client and [D] is the total number of samples in the entire FL system. Benefiting from the heterogeneous aggregation algorithm, the aggregated global model retains knowledge from all clients, enabling to perform robustly across domains.
The overall learning process of our proposed DapperFL is described in Algorithm 2
4 Experimental Evaluation
4.1 Experimental Setup
Implementation Details. We implement DapperFL with real-world FL platform FedML 449 with deep learning tool PyTorch [50]. We build the FL environment with a client set containing 10 heterogeneous edge devices and central server on the FedML platform. Following convention setting [5] 25 51), we categorize these heterogeneous devices into levels according to their system capabilities, i.e., C (1 E [1,5)) 5)), where C represents device set belongs to level 1. The system capabilities of devices in set C decrease as level increases, Our experiments are conducted on GPU server with NVIDIA RTX 3080Ti GPUs. Each experiment is executed three times with three fixed random seeds to calculate average metrics and ensure the reproducibility of our results.
Datasets and Data Partition. We evaluate DapperFL two domain generalization benchmarks, Digits (52 53 54 (55) and Office Caltech (56) that are commonly used the literature for domain generalization. The Digits consists of the following four domains: MNIST, USPS, SVHN and SYN. The Office Caltech consists of the following four domains: Caltech, Amazon, Webcam, and DSLR. For each benchmark, while guaranteeing that each domain contains at least one client and that each client only belongs to one domain. To generate non-IID local data, we randomly extract a proportion of data from the corresponding domain as the statistical characteristics vary among domains. Following the conventional data partition setting [15), we set 1% as the local data proportion of Digits and 20% as that in Office Caltech based on the complexity and scale of the benchmarks.
distribute the simulated 10 clients randomly each domain
Models.
We adopt ResNet10 and ResNet18 [57] the backbone models for Digits and Office Caltech, respectively. In the following evaluations, both the DapperFL and comparison frameworks train the models from scratch for fair comparison.
Comparison Frameworks. We compare DapperFI. with state-of-the-arr (SOTA) FL frameworks, including FedAvg [3) MOON [16), FedSR [14], FPL [15), FedDrop [0], FedProx [17), FedMP []].
7

---

Table 1: Comparison of model accuracy (%) on Digits.
System Heter.
Global
FL frameworks
MNIST
USPS
SVHN
SYN
accuracy
FedAvg
3
X
95.89(1.47)
86.84(0.80)
78.39(3.24)
33.63(2.87)
(0.6)
MOON
16
x
93.03(1.97)
78.38(5.81)
84,45(7.55)
25.97(3.28)
69,44(0.53)
FedSR 4
x
96.77(0.73)
86.15(2.38)
81.48(1.77)
31.64(0.40)
73.89(0.57)
FPL [15
x
95.54(1.78)
87.69(0.98)
83.74(4.26)
34.73(1.53)
74.17(0.95)
FedDroo
89.48((.56)
82.51(1.17)
72.98(0.83)
29.35(1.97)
66.85(0.93)
FedProx
96.68(0.96)
83.96(0.73)
76.69(3.50)
30.95(1.42)
70.74(0.52)
FedMP |
94.16(3.32)
85.30(2.66)
72.29(0.89) 35.12(2.00) 81.37(1.92)
NeFL [12
84.98(1.07)
88.49(4.17)
78.41(2.33)
36.02(5.72)
67. 07.64(0.30)
DapperFL (ours)
96.25(2.10)
86.30(1.24)
82.45(1.72)
37.26(2.71)
74.30(0.26)
Table 2: Comparison of model accuracy (%) on Office Caltech.
System Heter.
Global accuracy
FL frameworks
Caltech
Amazon
Webcam
DSLR
FedAvg
X
66.07(2.46)
76.84(3.18)
65.52(4.98)
56.67(1.98)
64.54(1.10)
MOON 16
x
75.79(1.69) 65.62(3.74)
53.33(1.93) 72.41(2.63)
61.86(0.79)
FedSR 14
x
62.95(2.25)
78.95(3.29)
75.86(3.59)
50.00(3.34)
65.47(1.13)
FPI. 15
X
63.84(3.17)
82.63(4.11)
65.52(2.63)
60.00(3.85)
65.45(1.15)
FedlDrop 10
V
66.07(0.89)
79.47(2.30)
56.90(3.98)
53.33(6.94)
60.58(1.42)
FedProx 17
V
61.61(4.09)
71.05(4.98)
68.97(4.98)
46.67(1.93)
62.08(1.11)
FedMP 11
65.62(2.49)
75.79(2.43)
56.90(3.59)
66.67(3.34)
62.34(0.93)
NeFL [12]
54.91(1.57)
71.05(1.61)
77.59(4.56)
66.67(3.85)
62.26(1.34)
DapperFL (ours)
64.73(1.03)
81.58(3.29)
74.14(1.99)
66.67(3.85)
67.75(0.97)
and NeFL 12]. These FL frameworks are either classical or focus on addressing system heterogeneity or domain shifts issues. The comparison frameworks are described detail in Appendix
Default Hyper-parameters. To conduct fair evaluations, the global settings and local training hyper- parameters of FL. are identical values to both the DapperFI. and comparison FI frameworks. For the framework-specific The tlyper-paammee used in our evaluations are described in Appendix B
we use their default settings without changing them.
Evaluation Metrics. In this paper, we evaluate the performance of the global model using the average Top-1 accuracy across all domains. In evaluating the resource consumption of the clients, adopt both the total number of parameters and the Floating- PPint Operations (FLOPs) of the local model.
4.2 Performance Comparison
Model Performance Across Domains. Tables and 2 respectively provide detailed analyses of model accuracy on the Digits and Office Caltech benchmarks. The term "System Heter." indicates whether the respective framework supports system heterogeneity The results reported model accuracy with corresponding standard deviations in brackets.
As shown in both Tables I and 2 DapperFL achieves the highest global accuracy on both the Digits and Office Caltech benchmarks compared with the comparison frameworks. On Digits and Office Caltech, DapperFL's global accuracy is O. 1.13% and 2.28% better than the runner-up, respectively. This indicates that the global model learned by DapperFL. more robust across all domains and therefore possesses better domain generalization. Moreover, DapperFL always achieves competing accuracy on individual domains and even the best domain accuracy on the SYN, Amazon, and DSLR. This demonstrates that the DAR module of DapperFL implicitly encourages the encoder to learn invariant representations, resulting in DapperFL. being unlikely to over-fit on the specific
domain- Gomain-invariant
NNoe that when evaluating frameworks that do not support heterogeneous clients, we allow all clients to participate FL by disregarding resource constraints clients.

---

domain(s) and possessing better domain generalization ability. Furthermore, DapperFL tolerates heterogeneous clients through personalized pruning ratio p. achieving adaptive resource consumption reductions of (20%, 40%, 60%, 80%] for clients in {C2. Ca, Ca, Cs while outperforming comparison FL. frameworks that support heterogeneous clients. This is evidence that the MFP module of DapperFI. effectively reduces the resource consumption of clients with domain shifts. In addition, we provide learning curves for both global and domain accuracies in Appendix C a comprehensive analysis.
Impact of Model Footprints. In heterogeneous
100
100
FL, resource constraints on the clients often re- sult in limited model footprints (i.e., number of parameters and FLOPs). To investigate the impact of the model footprints on model per- formance, we compare the DapperFl. with two
Model Accuncy
80
FeMM
Motel Acaney
Not.
ReMP
60
60
40-
30
20
SOTA FL frameworks (i.e., FedMP and NeFI.) that are equipped with adaptive pruning tech- niques. The pruning ratio p determines both the reduction in the number of parameters and the FLOPs, We maintain consistent for every device, with predetermined values selected from the set (0.2, 0.4, 0.6, 0.8).
0.2
0.4
0.6
03
0.2
0.4
0.6
0.8
Proning
Pruning
(a) Digits
(h) Office Caltech
Figure 3: Comparison of model accuracy of
FedMP, NeFL, and DapperFL with different prun- ing ratios on the Digits and Office Caltech.
Figure illustratte the comparative results under varying pruning ratios on both Digits and Office Caltech. As depicted, these frameworks tend to achieve higher accuracy with smaller pruning ratios, and DapperFI. consistently outperforms the others across all values on both benchmarks. This underscores the superior adaptability of DapperFL. in heterogeneous FL environments. Notably, in Figure ( the accuracy of DapperFL at = 0.4 surpasses its accuracy at p = 0.2. This discrepancy arises from the ver-paraaeeterization of ResNet18 for the classification task in Office Caltech. The improved accuracy observed at higher pruning rates suggests that the MFP module of DapperFL. effectively prunes unnecessary parameters while enhancing model generalization rather than compromising it. This also demonstrates the reason behind DapperFL's superiority over comparison frameworks that employ full-size models. Furthermore, comprchensively evaluate the effect of DapperFL's pruning ratio p on model performance in Appendix D
4.3 Ablation Study
Effect of Key Modules in DapperFL. We evaluate the performance of DapperFL with and without the MFP and DAR modules, individually and in combination. The fol- lowing configurations are considered: "DapperFL. w/o MFP+DAR", DapperFL. without the MFP and DAR mod- ules. "DapperFL w/o DAR". DapperFL without the DAR module. "DapperFL w/o MFP", DapperFL without the MFP module. "DapperFL", the complete DapperFL frame- work. It is noteworthy that when the MFP module is not employed in DapperFL,, it performs model pruning using & norm on the local models directly.
Table 3: Effect of DapperFL's Key Mod- ules on Digits and Office Caltech.
Configuration
Digits
Office
DapperFL w/o MFP+DAR
71.94%
62.65%
DapperFL w/o DAR
72.37%
64.88%
DapperFL w/o MFP
73.34%
66.28%
Table summarizes the ablation study results on both benchmarks. On Digits, the results indicate a gradual improvement in accuracy as we incorporate the MFP and DAR modules into DapperFL. DapperFL achieves the highest accuracy at 74.30%, highlighting the synergistic effect of both MFP and DAR in enhancing model performance. Similarly, observe consistent pattern on Office Caltech. DapperFL w/o MFP+DAR yields the lowest accuracy, while the inclusion of both MFP and DAR results in steady improvement. The complete DapperFL framework attains the highest accuracy at 67.75%, reinforcing the complementary roles played by MFP and DAR in bolstering domain adaptability in heterogeneous FL.
DapperFL
74.30%
67.75% %
Effect of Hyper-Parameters within the MFP Module. We investigate the influence of the hyper- parameters coo O'min and e within the MFP module on the overall performance of DapperFI.. In this experiment, only the hyper-parameter under evaluation is modified, while others retain their default settings outlined in Appendix B

---

Augunay (88
76
Model Accuracy (8)
40)
A
Model Accuracy
66
66
30
67
Mosgy
61
Digiis Offace
AVG
Dipis
AVC
Dights
-+-VVG
Digins
-Offiie -VAV
0.001
0.005
no1
005
(a) Effect of on in MFP
(b) Effect of amnin in MFP
(c) Effect of in MFP
(d) Effect of y in DAR
Figure 4: The effect of plyper-parmeenn in the MFP and DAR modules model accuracy. "Office" in the legend represents the Office Caltech benchmark, and the average accuracy calculated for both benchmarks is labeled as "AVG".
Figure 4aa illustrates the effect of og on model accuracy. The og parameter dictates the initial weight of the global model fused into the local model. In both benchmarks, the optimal og is evident at 0.9, resulting in the highest average accuracy. This result underscores the significance of infusing the local model with knowledge from other domains early in the FL process to enhance its generalization. Figure describbs the effect of Ommn on model accuracy, where amin determines the minimum weight of the global model fused into the local model. In both benchmarks, the optimal average accuracy achieved when aman set to 0.1. This finding signifies that, as the FL process proceeds, the local model benefits from acquiring more domain-speciff knowledge to uphold its personalization.
(ec))/4ccooeereee the effect of on model accuracy. The parameter controls the rate at which the fusion factor dg diminishes towards Omin- In both benchmarks, the optimal is discernible at 0.2, resulting in the highest average accuracy. This observation emphasizes that the most effective rate of decrease for the fusion factor oo towards Ohmin is achieved when 0.2 in both benchmarks. Furthermore, recognizing the abnormal trends the relationship between model accuracy and hyper-parameter when it is less than 0.2, we implement Bayesian search as an automatic selection mechanism to find the optimal value e. The results are provided in Appendix B
Effect of Sayeee-arraeetr within the DAR Module. Figure provides configurations with varying y values across both Digits and Office Caltech. The results show an increase in accuracy with higher y values until 0.01 on Digits, where the highest accuracy of 74.30% is achieved. notable drop in accuracy is observed at y 0.1, indicating that excessively large regularization may hinder model performance on Digits. Similar to Digits, an increase in accuracy is observed with higher y values until 7 = 0.01, reaching the highest accuracy of 67.75% The impact of regularization is more pronounced on Office Caltech, as seen by the significant drop in accuracy y 0.1. This suggests that the regularisation factor y in the DAR Module plays crucial role in solving the domain shift problem in FL and careful choice of can improve the model performance.
Conclusion
We have described our proposed FL framework DapperFL. tailored for heterogeneous edge devices with the presence of domain shifts. Specifically, DapperFL. integrates dedicated MFP module and a DAR module achieve domain generalization adaptively in heterogeneous FL. The MFP module addresses system heterogeneity challenges by shrinking the footprint of local models through personalized pruning decisions determined by both local and remaining domain knowledge. This effectively mitigates limitations associated with system heterogeneity. Meanwhile, the DAR module introduces a specialized regularization technique to implicitly encourage the pruned local models to learn robust local representations, thereby enhancing DapperFL's overall performance. Furthermore, we designed a specific aggregation algorithm for DapperFL to aggregate heterogeneous models produced by the MFP module, aiming to realize robust global model across multiple domains. The evaluation of DapperFL's implementation on a real-world FL platform demonstrated that outperforms several SOTA FL frameworks two benchmark datasets comprising various domains.
Limitations and Future Work. Despite its potential addressing system heterogeneity and domain shifts, DapperFL. introduces four hyper-parameters og. Omin, and r associated with the DG performance of the global model. A potential future direction involves the automatic selection of these paremeerraeterss, therefore enhancing the flexibility and accessibility of the DapperFL.
10

---

Acknowledgments and Disclosure of Funding
This research supported part by the National Natural Science Foundation of China No. 92267104 and No. 62372242. Dr. Xuyun Zhang is the recipient of an ARC DECRA (project No. DE210101458) funded by the Australian Government. The work of K.-K. R. Choo supported only by the Cloud Technology Endowed Professorship
References
[1] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang. Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: comprehensive survey. IEEE Communicutions Surveys & Tutorials, 22(3):2031-206: 2020.
[2] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. Foundations and trends@ in machine learning, ((1-2))1-216 2021.
[3] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication cettrummiiatto-eeffiiitnt efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[4] Dashan Gao, Xin Yao, and Qiang Yang. A survey on heterogeneous federated learning. arXiv preprint arXiv:2210.04505, 2022.
[5] Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Model-heterogennouss federated learning with rolling sub-model extraction. In Advances in Neural Information Processing Systems, volume 35, pages 29677-29690. Curran Associates, Inc., 2022.
[6] Liping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi, and Han Yu. Fedgh: Heterogeneous federated learning with generalized global header. In Proceedings of the 31st ACM International Conference on Multimedia, pages 8686-8696, 2023.
[7] Liling Zhang, Xinyu Lei, Yichun Shi, Hongyu Huang, and Chao Chen. Federated learning with domain generalization. arXiv preprint arXiv:2111.1042 2021.
[8] Ruqi Bai, Saurabh Bagchi, and David Inouye. Benchmarking algorithms for federated domain generalization. In The Twelfth International Conference on Learning Representations, 2023.
[9] Jungwuk Park, Dong-Jun Han, Jinho Kim, Shiqiang Wang, Christopher Brinton, and Jaekyun Moon. Stablefdg: Style and attention based learning for federated domain generalization. Advances Neural Information Processing Systems, 36, 2024.
[10] Sebastian Caldas, Jakub Konečny, Brendan McMahan, and Ameet Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint 2018.
[11] Zhida Jiang, Yang Xu. Hongli Xu, Zhiyuan Wang, Chunming Qiao, and Yangming Zhao Fedmp: Federated learning through adaptive model pruning in heterogencous edge computing. In 2022 IEEE 38th International Conference Data Engineering (ICDE), pages 767-779,
2022.
[12] Honggu Kang, Scohyeon Cha. Jinwoo Shin, Jongmycong Lee, and Joonhyuk Kang. Nefl: Nested federated learning for heterogeneous clients. arXiv preprint arXiv:2388.07700 2023.
[13] Yabin Zhang, Minghan Li, Ruihuang Li. Kui Jia, and Lei Zhang. Exact feature distribution matching for arbitrary style transfer and domain generalization. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 8035-8045, 2022.
[14] A Tuan Nguyen, Philip Torr, and Ser Nam Lim. Fedsr: A simple and effective domain generalization method for federated learning. Advances Neural Information Processing Systems, 15:38831-38843 2022.

---

[15] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 16312-16322. IEEF, 2023.
[16] Qinbin Li, Bingsheng He, and Dawn Song. Monel-contrastive federated learning. In Proceedings of the IEEF/CVF conference on computer vision and paltern recognition, pages 10713-10722 2021.
[17] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 450294400040 2020.
[18] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprini urXiv:2010.01264, 2020.
[19] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou, Data-free knowledge distillation for heteroge- neous federated learning. In International conference on machine learning, pages 12878-12889. PMLR, 2021.
[20] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8432-8440, 2022.
[21] Ang Li, Jingwei Sun, Pengcheng Li, Yu Pu, Hai Li, and Yiran Chen. Hermes: an efficient federated learning framework for heterogeneous mobile clients. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, pages 420-437, 2021.
[22] Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K. Leung, and Leandros Tassiulas, Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, pages 1-13, 2022.
[23] Haiyan Zhao and Guodong Long, One-shot pruning for fast-adapting pre-trained models on devices. arXiv preprint urXiv:2307.04365, 2023.
[24] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 3133- PMLR, 2020.
[25] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Information Processing Systems, volume 33, pages 7611-7623. Curran Associates, Inc., 2020.
[26] Zachary Charles, Kallista Bonawitz, Stanislav Chiknavaryan, Brendan McMahan, et al. Fed- erated select: primitive for communication-and memory-efficiett federated learning. arXiv preprint arXiv:2208.00 09432, 2022.
[27] Hanhan Zhou, Tian Lan, Guru Prasadh Venkataramani, and Wenbo Ding. Every parameter matters: Ensuring the convergence of federated learning with dynamic heterogeneous models reduction. Advances in Neural Information Processing Systems, 36, 2024.
(28) Chandra Thapa, Pathum Chamikara Mahawaga Arachchige, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8485-8493, 2022.
(29] Wen Wu, Mushu Li, Kaige Qu, Conghao Zhou, Xuemin Shen, Weihua Zhuang, Xu Li, and Weisen Shi. Split learning over wireless networks: Parallel design and resource management. IEEE Journal on Selected Areas in Communications, 41 (4)::0511-1066 2023.
[30] Ngoc Duy Pham, Alsharif Abuadbba, Yansong Gao, Tran Khoa Phan, and Naveen Chilamkurti. Binarizing split learning for data privacy enhancement and computation reduction. IFEE Transactions Information Forensics and Security, 2023.
12

---

[31] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4396 4415, 2022.
[32] Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[33] Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain gen- eralization via entropy regularization. Advances in neural information processing systems, 33:16096-16109 2020.
[34] Ziqi Wang, Marco Loog, and Jan Van Gemert. Respecting domain relations: Hypothesis invariance for domain generalization. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9756-9763. IEEE, 2021.
[35] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[36] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularization. Advances in neural information processing systems, 31, 2018.
[37] Shangyu Chen, Wenya Wang, and Sinno Jialin Pan. Cooperative pruning in cross-domain deep neural network compression. In IJCAI, pages 2102-2108, 2019.
[38] Yuyang Zhao, Zhun Zhong, Fengxiang Yang, Zhiming Luo, Yaojin Lin, Shaozi Li, and Nicu Sebe. Learning to generalize unseen domains via memory-based multi-source meta-learning for person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and 2021.
pattem recognition, pages
[39] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021.
[40] Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and LINGYU DUAN. Uncertainty modeling for out-of -iitt---ittiittionn generalization. In International Conference on Learning Representations, 2022.
[41] Wenke Huang, Mang Ye, and Bo Du Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10143-10153, 2022.
[42] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76 76, 2020.
[43] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1400, 2016
[44] Jason Yosinski. Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[45] Jaanqing Zhang, Yang Hua, Jian Cao, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Eliminating domain bias for federated learning in representation space. Advances in Neural Information Processing Systems, 36, 2024.
[46] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform- aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2820-288088888 2022-828. 2019.
[47] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. Proceedings of the IEEE international conference computer vision, pages 2736-2744, 2017.
13

---

[48] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
[49] Chaoyang He, Songze Li, Jinhyun So, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr. Fedml: A research library and benchmark for federated machine learning. arXiv preprint mrXiv:2007.13516 2020.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[51] Yiqun Mei, Pengfei Guo, Mo Zhou, and Vishal Patel. desource-adaptive federated learning with all-in-one neural composition. Advances Neural Information Processing Systems, 2022.
[52] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 ():27727772,,, 1998.
[53] Jonathan Hull. database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-5544 1994.
[54] Netzer Yuval. Reading digits in natural images with unsupervised feature learning. In Proceed- ings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[55] Prasun Roy, Subhankar Ghosh, Saumik Bhattacharya, and Umapada Pal. Effects of degradations on deep neural network architectures. arXiv preprint arXiv: 10108, 2018.
[56] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference computer vision and pattern recognition, pages 2066-2073. IEEE, 2012.
[57] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
14

---

A Introduction to Comparison Frameworks
The comparison FL frameworks used in this work are introduced as follows:
FedAvg is classical FL. framework that is widely adopted to aggregate distributed local models. During the aggregation period of FedAvg, the clients upload the updates to the server, and the central server responsible for generating the global model by performing weighted averaging on these local updates.
FedAvg (3:
FedDrop reduces the computational overhead of local updating and communication costs by generating compact models for clients. In FedDrop, the server adopts lossy compression techniques to compress the global model with uniform compression ratio for all clients. To satisfy the resource requirements of low-level clients, FedDrop needs to choose a large pruning ratio to compress the global model.
FedD [0]:
FedProx 17: : FedProx introduces a proximal term into the objective function, aimed at incentivizing
participants to uphold similarity with the global model throughout local training, meanwhile affording low-capacity clients the latitude to exccute fewer local updates.
MOON 16: MOON is a simple yet effective FL framework tailored for non-lID local data among clients. It capitalizes on model representation similarities to enhance the local training of individual clients through model-level contrastive learning.
FedMP employs adaptive local model pruning mechanism in each FL communication round to address the diverse resource constraints present on client devices. Simultaneously, leverages Residual Recovery Synchronous Parallel (R2SP) scheme aggregate models from heterogeneous clients.
FedMP [1]]
FedSR incorporates two regularization techniques implicitly align representations across domains in the FL environments. These regularizers implicitly align the marginal and condi- tional distributions of the representations, thereby enhancing domain generalization.
FedSR [4]
NeFL employs combination of depthwise and widthwise scaling techniques to partition model into submodels. Its primary objective is to enable cooource-constrinne clients to seamlessly engage in the Fl. process, meanwhile facilitating the training of models on more extensive datasets. To mitigate the challenges associated with training multiple submodels featuring varying architectures, NeFL, introduces a parameter decoupling mechanism.
NeFL 12
Table 4: Default hyper-parameters used in our evaluations.
Type
Hyper-parameter
Default value
Communication round
100
Global
Client number
10
settings of FL
Client participation rate
100%
Local epoch
Batch size
64
Local training of FL
Learning rate Momentum
0.01
0.9
Weight decay
1 x 10-5
in FedProx μ MOON
0.1
in MOON L2R in FedSR CMI in FedSR in FPL 0g in DapperFL
0.5
Framework- specific
0.01
0.001
0.02
0.9
Omin in DapperFL in DapperFL
0.1
0.2
y in DapperFL
0.01
15

---

FPL aims to learn robust global model in FL with domain shifts. It improves generaliza- tion while maintaining discriminability by leveraging contrastive learning with cluster prototypes. Additionally, FPL utilizes unbiased prototypes to provide fair and stable convergence point.
FPL [5]
B Default Hyper-parameters
The default hyper- paper-paraeetess used in our evaluations are described Table
C Learning Curves
We provide learning curves of global accuracy and domain accuracy in Figure and Figure 6 respectively. These curves provide the maximum Top-1 accuracy achieved until each communication round of FL.
Digits
Office Caltech
80
80
Global Accuracy (%)
Global Accuracy (%)
60
74
40
40
20
62
66
100
0
25
50
75
100
0
25
50
75
100
Communication Round
Communication Round
FedAvg
FedDrop
FedProx
MOON
FedMP
FedSR
NeFl.
FPL
Dappert
Figure 5: Learning curves of global accuracies for FedAvg, FedDrop, FedProx, MOON, FedMP,
FedSR, NeFL, FPL, and DapperFL on the Digits benchmark and the Office Caltech benchmark.
MNIST
USPS
SVHN
SYN
Domuin Accuracy ($) Domain Accuracy
900
90
80
80
80
30
70
70
70
20
60
60
60
50
75
100
25
50
75
25
50
75
100
50
100
Caltech
Amazon
W Webcum
DSLR
80
70
50
70
50
50
40
60
30
40
30
50
10
50
75
100
25
50
75
100
25
50
75
100
50
100
Communicution Round
Communication Round
Communication Round
Communication Round
FedAvg
FedDrop
FedProx
MOON
PeaMP
FedSR
NeFL
PPL
DapperFFL
Figure 6: Learning curves of domain accuracies for FedAvg, FedDrop. FedProx, MOON, FedMP, FedSR, NeFL, FPL, and DapperFL on the Digits benchmark and the Office Caltech benchmark.
D Effect of Pruning Ratio p
We systematically investigate the impact of varying pruning ratios (p) on the model performance of DapperFI. using both the Digits and Office Caltech benchmarks. Pruning ratios ranging from 0.2 to 0.8 are considered, representing different degrees of model compression. Table presents
presents the
16

---

model footprint and accuracy of DapperFL under different pruning ratios on the Digits benchmark. As the pruning ratio increases, the number of parameters (#Para) and FLOPs decrease, reflecting the expected reduction in model size and computational complexity. Notably, the model maintains competitive accuracy when pruning ratios increase from 0.2 to 0.6, indicating the robustness of DapperFL. to varying degrees of model compression. Similarly, Table T details the model footprint and accuracy of DapperFL under different pruning ratios on the Office Caltech benchmark. Consistent with the Digits benchmark results, the model adapts effectively to increased pruning from 0.2 to 0.6, with reductions in both parameters and FLOPs.
In summary, DapperFL consistently outperforms the baseline frameworks that utilize the entire model, such FedAvg and MOON, as illustrated in Table and T2ble[]] when employing a pruned model with pruning ratio ranging from 0.2 to 0.6. Despite incurring an inevitable accuracy loss at 0.8 on both benchmarks, DapperF1. still outperforms FedMP and NeFI., both equipped with adaptive pruning capabilities, as illustrated in Figure 3 These observations collectively demonstrate the robustness of DapperFL. across a spectrum of pruning ratios, emphasizing its potential for deployment in resource-oonstrainnd FL environments. The ability to achieve substantial model compression (with the pruning ratio p ranging from 0.2 to 0.6) without significant loss in accuracy makes DapperFL promising solution for real-world applications where resource efficiency is paramount.
Table 5: Model footprint and accuracy of DapperFL under varying pruning ratios Digits.
Pruming ratio
#Para
FLOPs
MNIST
USPS
SVHN
SYN
Global accuracy
0.2
3.92M
203.34M
94.86%
83.36%
85.55%
32.84%
73.06%
0.4
2.94M
152.50M
89.42%
80.77%
84.13%
35.57%
71.76%
0.6
1.96M
101.67M
91.79%
82.16%
77.59%
29.65%
69,27%
0.8
0.98M
50.83M
63.38%
66.17%
58.38%
21.64%
48.14%
Table Model footprint and accuracy of DapperFL under varying pruning ratios on Office Caltech.
Pruning ratio p
#Para
FLOPPs
Caltech
Amazon
Webcam
DSLR
Global accuracy
0.2
8.94M
166.
68.30%
80.53%
63.79%
60.00%
66.80%
0.4
6.70M
274.60M
70.09%
79.47%
67.24%
56.67%
67.71 %
0.6
4.47M
183.06M
58.48%
80.00%
67.24%
50.00%
61 61.0%%
0.8
2.23M
91.53M
43.75%
53.16%
31.03%
33.33%
38.28%
E
Automatic Selection Mechanism for Hyper-parameter
We run DapperFL the Office Caltech benchmark 40 times, adopting distinct of less than 0.2 each time. The values are selected using the Bayesian search. The results are presented Fig.
68
67
Model Accuracy (%)
65
64
63
62
61
0.10
0.11
0.12
0.13
0.14
0.15
0.16
0.17
0.18 0.19
0.20
6
Figure 7: Model accuracy of DapperFL on the Office Caltech for different searched by the Bayesian algorithm. The maximum model accuracy as increases is plotted with the blue line for analysis.
17

---

As illustrated, the Bayesian search-basee automatic selection mechanism indicates that model ac- curacy is likely reach higher level when e approaches 0.2, aligning with our default setting of <= 0.2. It is noteworthy that, owing to the nature of Bayesian search, the sampled values are not uniformly distributed, as they tend to bias towards the optimal that maximizes model accuracy.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Abstract and Introduction sections accurately reflect the contribution and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The discussion about the limitations of this work is provided in the Conclusion
section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: This paper provides URL link to our released code and describes the training the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper provides URL link released code. The public datasets used in this paper are properly referenced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
Sublic//uiddes/Codeuubbisss
caaessionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppplcccuua.eeaeaecccuea
Stteee///ooSSSeesssSioi.a www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper describes the experimental setup in the Experimental Evaluation section and provides the detailed training and test details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: accuracy.
The main results are accompanied by the standard deviations of the model
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper describes the experimental environment in the Experimental Evalu- ation section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httiiissttttt////tt httpp////eaaapp....cpp/p//clllca
Answer: [Yes]
Justification:
This paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The assets used in this paper properly credited. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "JdmzySlbvM", "reviewer": "Reviewer_1ipw", "summary": "The paper introduces a novel federated learning framework to address system heterogeneity and domain shifts in edge computing environments. The framework employs a Model Fusion Pruning (MFP) module to generate personalized compact local models and a Domain Adaptive Regularization (DAR) module to enhance performance across multiple domains. Experimental results demonstrate the effectiveness of the proposed method in terms of accuracy improvements and model compression.", "questions": "1. Any suggestions when choosing hyperparameters (α0, αmin, ϵ, γ) for good global model performance in DapperFL? \n\n2. Why does the smaller model generated by DapperFL have an accuracy comparable to that of SOTA, and sometimes even exceed the accuracy of a full-size model? \n\n3.  In eq.6, the authors combine “local knowledge” and “global knowledge” to recover the local model. It could be better to introduce these definitions more clearly before using them. What do “local knowledge” and “global knowledge” in DapperFL stand for, and where are they derived from?", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is an innovative approach in edge environments. The framework’s ability to simultaneously tackle heterogeneous and low-resource challenges in edge environments. The proposed solution is highly applicable and effective in real-world edge computing scenarios. \n\n2. The paper’s contents are well-organized. The motivation is clearly illustrated and well supported by the proposed method. \n\n3. The experimental analysis is comprehensive.  For example, the chosen benchmark datasets include MNIST, USPS, SVHN, SYN, Caltech, Amazon, Webcam, and DSLR.  The comparisons with 8 SOTA frameworks add credibility to the claims. \n\n4. The proposed method is technique sounds. Moreover, the source codes are provided to ensure the reproducibility of this work.", "weaknesses": "1. The proposed method is sensitive to hyperparameter selection. For example, the framework introduces several hyper-parameters (α0, αmin, ϵ, γ) that significantly influence performance, which also complicates the application of DapperFL. As pointed out in the “Limitations” of this manuscript, it could benefit from providing a set of default values for these hyper-parameters or an automatic hyper-parameter selection mechanism. \n\n2. Experimental Results Analysis: Despite DapperFL achieving competitive model accuracy with a significantly smaller model footprint, it would be beneficial to discuss why a smaller model could outperform a larger one. \n\n3. It is unclear how the proposed meethod to be applied to new devices in test or production phase.\n\n4. Missing some related work, such as heterogenous FL [1,2] and model pruning on devices [3,4].\n\n[1] Zhuangdi Zhu, et al., Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning (pp. 12878-12889). ICML 2021.\n\n[2] Yue Tan, et al., Federated Prototype Learning across Heterogeneous Clients, AAAI 2022\n\n[3] Yuang Jiang, et al., Model Pruning Enables Efficient Federated Learning on Edge Devices, IEEE TNNLS 2022\n\n[4] Haiyan Zhao, et al., One-Shot Pruning for Fast-adapting Pre-trained Models on Devices, arXiv preprint arXiv:2307.04365", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. Below we address your concerns in order.\n\n## Response to Weakness 1 \\& Question 1:\nThanks for your suggestion to provide default values or an automatic selection mechanism for hyper-parameters. We acknowledge that hyper-parameter selection is indeed crucial, especially for practitioners looking to effectively apply the DapperFL framework. \n\nIn DapperFL, $\\alpha \\in [\\alpha_{min}, \\alpha_0]$ control the balance between local and global information, which can impact how well the model generalizes across different domains. The sensitivity factor $\\epsilon$ determines the rate at which $\\alpha$ decreases from $\\alpha_0$ to $\\alpha_{min}$, while $\\gamma$ influences the regularization strength.\nIn our paper, we have conducted ablation experiments (in Section 4.3) to identify suitable hyper-parameter settings. The experimental results suggest that: \n1) Increasing $\\alpha_0$ or decreasing $\\alpha_{min}$ generally enhances the incorporation of global knowledge, improving generalization. In our experiments, setting $\\alpha_0$ to 0.9 and $\\alpha_{min}$ to 0.1 resulted in good model accuracy.\n2) As $\\epsilon$ increases, model accuracy suffers from a decrease due to insufficient guidance from the global knowledge except for $\\epsilon$ is less than 0.2. \n3) $\\gamma$ helps regularize the model and prevent overfitting. In our experiments, setting it to an optimal value of 0.01 can balance model personalization on the local domain and generalization across multiple domains.\n\nFurthermore, recognizing the abnormal relationship between model accuracy and hyper-parameter $\\epsilon$ as it is less than 0.2, we implement Bayesian search as an automatic selection mechanism to find a better $\\epsilon$. Due to limited rebuttal time and hardware constraints, we have tried our best to include as many single runs as possible to search for a better $\\epsilon$ for DapperFL. Specifically, we run DapperFL on the Office Caltech benchmark 40 times, adopting a distinct $\\epsilon$ of less than 0.2 each time. The values are selected using the Bayesian search algorithm. The results are presented in Fig. 2 of the rebuttal PDF file. As illustrated, the Bayesian search-based automatic selection mechanism indicates that model accuracy is likely to reach a higher level when $\\epsilon$ approaches 0.2, aligning with our original paper's default setting of $\\epsilon=0.2$.\n\n\n## Response to Weakness 2 \\& Question 2:\nThe relationship between model accuracy and model footprint is complex and not necessarily linear, as evidenced by previous studies [1, 2]. In our work, we employ the MFP module within DapperFL, which addresses the over-fitting issues that can arise from the over-parameterization of neural networks. This module not only reduces the model footprint but also enhances the model's generalization capabilities, leading to more robust performance across diverse domains. Additionally, the DAR module is incorporated to further improve model accuracy, particularly in scenarios involving distributed clients with domain shifts.\n\n> [1] Z. Liu, et al. Learning efficient convolutional networks through network slimming, ICCV 2017.\n>\n> [2] S. Shen, et al. Efficient deep structure learning for resource-limited IoT devices, GLOBECOM 2020.\n\n\n## Response to Weakness 3:\nOne primary objective of DapperFL is to optimize a robust global model with strong domain generalization capabilities across distributed local data. In the test or production phase, this optimized global model can be deployed on new devices, ensuring good performance across different environments. Additionally, newly introduced devices can also participate in the DapperFL framework. They can contribute to the continual improvement of the global model by optimizing a local model based on their own local data, which is then aggregated to update the global model. This approach not only enhances the global model's adaptability to new data distributions but also leverages the unique data available on new devices to improve overall model robustness.\n\n\n## Response to Weakness 4:\nThank you for highlighting these relevant studies. These studies offer valuable insights into heterogeneous FL and model pruning on devices, aligning well with the themes of our research. We will incorporate these references into our paper to discuss their relationship to our study, highlighting similarities, differences, and how our work extends or diverges from these existing approaches.\n\n\n## Response to Question 3:\nIn our work, “local knowledge” refers to the feature extraction capabilities of the local model, which are derived from the specific data available in its local domain. This knowledge encapsulates the nuances and characteristics of the data that the local model has been trained on. \nOn the other hand, “global knowledge” represents the aggregated feature extraction capabilities of the global model, which are informed by data across all participating domains. The global model synthesizes diverse knowledge from different local models to provide a more generalized understanding that is applicable across multiple domains. By combining these two forms of knowledge, we aim to leverage both the specialized insights of local models and the generalized capabilities of the global model, thereby enhancing the performance and adaptability of DapperFL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "df0lPebR9t", "reviewer": "Reviewer_pq1G", "summary": "This paper proposes DapperFL, an innovative FL framework designed to enhance model performance across multiple domains within heterogeneous FL environments. DapperFL addresses the system heterogeneity challenge through the deployment of a dedicated Model Fusion Pruning (MFP) module. Additionally, a Domain Adaptive Regularization (DAR) module is introduced to improve the overall performance of DapperFL. The evaluation results are positive compared with SOTA.", "questions": "Q1: Why is the l_1 norm (rather than other normalization methods such as the l_2 norm) used to calculate mask matrices?\nQ2: When DapperFL aggregates heterogenous models on the central server, how does the sever know the architecture of these heterogenous models?", "limitations": "See weaknesses.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "S1: The proposed framework is novel in FL to apply local-knowledge-based heterogenous pruning to the problem of domain shift in FL.\nS2: This paper achieves better model performance with fewer resource consumptions on a real-world FL platform, which is a significant technical contribution to FL.\nS3: The paper is easy to follow, with a clear presentation and publicly available source code.", "weaknesses": "W1: It could be better to provide a discussion on why only one epoch is enough to fine-tune the global model in Algorithm 1.\nW2:\tIn the evaluation section, the authors should provide more framework-specific default values for hyper-parameters.\nW3: The subscript “i” is used to denote both client subscripts in the methods section and client-level subscripts in the experiments section. It would be better to use different notation.\nW4: The limitations of this work could be elaborated more to let the audience develop a comprehensive understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. Below we address your concerns in order.\n\n## Response to Weakness 1:\nWe utilize one epoch of local training to determine the pruned models for the following reasons: \n\n1) Additional local epochs do not significantly enhance the model's performance, which justifies the use of a single epoch for efficiency. As noted in [1], experiments have demonstrated that extending local training beyond one epoch yields results comparable to those achieved with just one epoch.\n\n2) In previous domain generalization-related FL research, such as [1], one epoch is also employed to collect local domain information. This method has proven adequate for capturing essential features and domain characteristics.\n\n3) Pioneering research in model design and neural architecture search, such as in [2], has demonstrated that a few epochs are sufficient to obtain a coarse estimate of a sub-model. This approach is effective in quickly assessing model configurations without extensive computation. \n\n\n>[1] Zhang, Jianqing, et al. Eliminating domain bias for federated learning in representation space. NIPS 2024.\n>\n>[2] Tan, Mingxing, et al. Mnasnet: Platform-aware neural architecture search for mobile. CVPR 2019.\n\n## Response to Weakness 2:\nWe have provided the default values for three framework-specific hyper-parameters in FedProx and MOON, as well as all four hyper-parameters in our DapperFL framework. Additionally, we have included more default values for the hyper-parameters in the comparison frameworks in the rebuttal PDF file, which will be incorporated into our final version. Specifically, the default values are $\\alpha^{L2R}=0.01$ in FedSR, $\\alpha^{CMI}=0.001$ in FedSR, and $\\tau=0.02$ in FPL. These values are consistent with those used in the original implementations of these frameworks, ensuring accuracy and reliability in our comparisons.\n\n## Response to Weakness 3:\nThank you for your meticulous observation. To avoid confusion and ensure clarity, we will change the notation for client-level subscripts in the experiments section to $l$. We hope this adjustment will improve the readability and precision of our paper.\n\n## Response to Weakness 4:\nWe acknowledge that further elaboration on the limitations of this work could provide the reader with a fuller understanding. As an example, the limitations of hyper-parameter selection will be described below:\n\n“Despite DapperFL shows promise in addressing system heterogeneity and domain shifts, it also introduces four hyper-parameters: $\\alpha_0$, $\\alpha_{min}$, $\\epsilon$, and $\\gamma$. These hyper-parameters influence the domain generalization performance of the global model. One limitation is the manual tuning required for these hyper-parameters, which may be time-consuming and require domain-specific expertise. As such, a potential future direction is the development of an automatic selection mechanism for these hyper-parameters. This enhancement would not only improve the flexibility of DapperFL but also make it more accessible to users without extensive knowledge in hyper-parameter tuning, thereby broadening the applicability and ease of deployment of our framework.”\n\n## Response to Question 1:\nWe opted for the $\\ell_1$ norm instead of the $\\ell_2$ norm for the following reasons: The $\\ell_1$ norm requires fewer computational resources, making it more suitable for our framework, which targets resource-constrained edge devices. \nAdditionally, as shown by pioneering work [3], there is no significant difference in the effectiveness of using $\\ell_1$ norm versus $\\ell_2$ norm for calculating mask matrices in the context of model pruning. This finding supports our choice, as the $\\ell_1$ norm offers a more resource-efficient alternative without compromising the quality of the pruning process.\n\n>[3] Li, Hao, et al. Pruning Filters for Efficient ConvNets. ICLR 2022.\n\n## Response to Question 2:\nIn our proposed framework, the heterogeneous models are transmitted to the server along with the mask matrices. These mask matrices indicate the architecture of the heterogeneous models. \nSpecifically, the mask matrices encode the structure of each model by highlighting which parts of the model are retained and which are pruned. \nThis information allows the central server to understand and manage the varying architectures of the heterogeneous models during aggregation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GZMaj7KeJI", "reviewer": "Reviewer_Ct3S", "summary": "The authors propose a new FL scenario that faces challenges of both system heterogeneity and domain shifts. This situation means that the clients have varying capabilities and their data domains also differ. The authors claim that they propose three novel modules to tackle the challenges: Model Fusion Pruning (MFP), Domain Adaptive Regularization (DAR), and a specific aggregation algorithm. The MFP module aims to address the challenge of system heterogeneity. It first fine-tunes the global model with local data to get a local model and then fuses the local model and the global model using a simple weighted average, where the fusion factor decays with the training epochs. Then, it prunes the fused model based on the L1 norm and continues training this model on the local data. The DAR module aims to address the challenge of domain shifts. It adds an L2 regularization on the first few layers of the fused model during the training process. To aggregate the locally pruned models, the models' structure will be discovered using the weights of the global model (of the last communication round) before global aggregation. The authors evaluate their method on two domain generation datasets, and the experimental results show that DapperFL achieves a SOTA performance.", "questions": "1. How do you segment the model in the Domain Adaptive Regularization module?\n2. Do you have any evidence showing the importance or prevalence of the scenario present in the Introduction section (FL with both system and domain heterogeneity)?", "limitations": "The authors discuss the limitation that the proposed method introduces four hyper-parameters and the choice of these hyperparameters significantly impacts the performance of the method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel scenario of Federated Learning (FL) where both the capability and data domains are heterogeneous among clients.\n2. Experimental evaluation indicates that the proposed method achieves SOTA performance. This paper also conducts an ablation study to show that each component in the design has a positive effect on the final results.", "weaknesses": "1. This paper lacks significant innovation. The proposed framework is just a combination of existing approaches to solving the problems of domain shifts and system heterogeneity, respectively. It’s common to use model pruning to solve the problem of system heterogeneity, such as [1][2][3]. The model aggregation method seems natural for updating the subnetwork of the local model on the global model and has already been proposed in similar work such as [3].\n2. The explanation of the proposed method is unclear. For example, the Domain Adaptive Regularization module does not specify how the model is segmented.\n3. The motivation of this paper is not strong enough. This paper lacks evidence to demonstrate the importance of the scenarios present in the Introduction section (with both system heterogeneity and domain heterogeneity).\n\n[1] Jiang, Z., Xu, Y., Xu, H., Wang, Z., Liu, J., Chen, Q., & Qiao, C. (2023). Computation and communication efficient federated learning with adaptive model pruning. *IEEE Transactions on Mobile Computing*, *23*(3), 2003-2021.\n[2] Li, A., Sun, J., Zeng, X., Zhang, M., Li, H., & Chen, Y. (2021, November). Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. In *Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems* (pp. 42-55).\n[3] Li, A., Sun, J., Li, P., Pu, Y., Li, H., & Chen, Y. (2021, October). Hermes: an efficient federated learning framework for heterogeneous mobile clients. In *Proceedings of the 27th Annual International Conference on Mobile Computing and Networking* (pp. 420-437).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. Below we address your concerns in order.\n\n\n## Response to Weakness 1:\n\nWe would like to emphasize that, our proposed framework is **NOT** a combination of existing approaches. \nWhile our work builds on established concepts, it introduces novel components and methods that address significant gaps in the current literature, particularly concerning domain shifts in resource-constrained edge environments. \nWe outline the substantial differences and innovative contributions of our DapperFL framework from existing works as follows: \n\n1) **Transfer Learning-Inspired Model Fusion Pruning.**\nExisting heterogeneous FL frameworks, such as [1,2,3], focus on addressing data heterogeneity (e.g., variations in sample and label distributions) but do not directly tackle the challenge of domain shifts, which can significantly impact the performance of pruned models. \nOur work introduces Model Fusion Pruning (MFP), which not only accounts for data heterogeneity but also addresses domain shifts. MFP dynamically combines local domain information with cross-domain information to prune local models effectively, enhancing their performance under domain shift conditions.\n\n2) **Lightweight Solutions for Domain Shifts.**\nCurrent methods often require extensive data collection transmission to handle domain shifts, which is impractical in resource-constrained environments. \nOur DapperFL framework, however, offers a novel resource-efficient approach to handling domain shifts through the MFP and Domain Adaptive Regularization (DAR) modules. The MFP module prunes local models leveraging local and cross-domain information. The DAR module introduces a regularization term that encourages the generation of domain-invariant representations at the local level, eliminating the need for transmitting domain-invariant representations, which is commonly required in existing studies. \n\n3) **Specific Model Aggregation in Cross-Domain Environments.**\nExisting works, including [3], lack mechanisms to ensure that domain-specific features are retained in heterogeneous environments.\nIn contrast, we propose a novel model recovery mechanism during heterogeneous model aggregation. This mechanism ensures the preservation of specific domain knowledge while transferring global knowledge, as detailed in Eq.6 of our original paper. This unique operation reinstates the pruned model's architecture and incorporates global knowledge, differentiating our approach from existing methods.\n\n>[1] Jiang, Z., et al. Computation and communication efficient federated learning with adaptive model pruning. TMC, 2023. \n>\n>[2] Li, A., et al. Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. SenSys 2021. \n>\n>[3] Li, A., et al. Hermes: an efficient federated learning framework for heterogeneous mobile clients. MobiCom 2021.\n\n\n## Response to Weakness 2 \\& Question 1:\n\nWe follow a conventional setting commonly used in representation learning research to segment the model. Specifically, all layers except the final linear layer act as the encoder, while the last linear layer of the model acts as the predictor. \nWe acknowledge the importance of clarity and will include a detailed description of this segmentation process.\n\n\n## Response to Weakness 3 \\& Question 2:\n\nConsider a mobile voice assistant like Apple Siri, which requires an efficient FL framework to collect and analyze user voice data [4]. Users may interact with Siri using different accents even when speaking the same language. This variation leads to domain shift problems, as the system must adapt to different speech patterns. Additionally, the diversity in mobile phone capabilities introduces system heterogeneity, as devices with varying processing powers may affect the performance of local computations. Another pertinent example is the IoV network, where smart vehicles collect, compute and communicate data for tasks like navigation and traffic management [3,5]. The data collected, such as images, can vary significantly in quality due to factors like motion blur, leading to multi-domain datasets. On the other hand, the varying computing power and network conditions of vehicles create system heterogeneity challenges [5]. Beyond these scenarios, our framework is relevant to various scenarios such as smart homes and smart health systems [1,2].\n\nWe have added Fig. 1 to the rebuttal PDF to visually demonstrate the impact of these issues on FL performance. Specifically: \n1) **System Heterogeneity:** Device 1, with stringent resource constraints, fails to complete its model update within the maximum allowed time. As a result, the central server excludes it from the FL process and misses out on the data features captured by Device 1. \n2) **Domain Shifts:** Devices 2 and 3 collect data from different domains, leading to diverse local models. With non-IID data, the global model aggregation may become biased, particularly if Device 3 contributes more data, thus potentially limiting the benefit for Device 2.\n \n>[1] Lim, Wei Yang Bryan, et al. Federated learning in mobile edge networks: A comprehensive survey. IEEE Commun. Surv. Tutorials 2020.\n>\n>[2] Nguyen, Dinh C., et al. Federated learning for Internet of Things: A comprehensive survey. IEEE Commun. Surv. Tutorials 2021.\n>\n>[3] M. Gerla, et al. Internet of Vehicles: From intelligent grid to autonomous cars and vehicular clouds, WF-IoT 2014.\n>\n>[4] Paulik, Matthias, et al. Federated evaluation and tuning for on-device personalization: System design \\& applications. arXiv 2021.\n>\n>[5] D. Ye, et al. Federated learning in vehicular edge computing: A selective model aggregation approach, IEEE Access 2020.\n\n\n## Response to Limitation 1:\n\nThank you for your insightful comments. Due to the rebuttal's length limitations, please refer to our **Responses to Weakness 1 \\& Questions 1** from **Reviewer 1ipw** for detailed information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "olNXYlSsEv", "reviewer": "Reviewer_npsw", "summary": "This paper proposes DapperFL, a novel framework for domain adaptive federated learning designed specifically for heterogeneous edge devices. It addresses the challenge of system heterogeneity and domain shifts in FL by integrating a Model Fusion Pruning (MFP) module and a Domain Adaptive Regularization (DAR) module. The framework also incorporates a novel aggregation approach to handle heterogeneous local models with varying architectures and weights.", "questions": "•\tIs there a practical difference between the fine-tuning global model steps inspired by the spirit of transfer learning in this article and the local model training after initializing the global model in regular federated learning? Can you provide a more specific explanation of the role of transfer learning in this step?\n\n•\tThe global model deployment step appears to result in the global model being stored on the local client and requiring fine-tuning as well as mask operations for traversing each neural unit. Does this contradict resource-constrained scenarios in heterogeneous systems?", "limitations": "•\tThe authors pointed out the limitations of the paper in terms of hyper-parameters selection. However, this article still has certain limitations in its solution for system heterogeneity. In addition, it is necessary to add theoretical conversion analysis of the proposed federated learning algorithm.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "•\tOriginality: The author focuses on simultaneously addressing system heterogeneity and domain shifts in the context of federated learning. While previous work has devoted to address these two aspects individually, there is a lack of solutions that tackle both problems simultaneously. DapperFL offers a unique combination of model pruning and domain adaptation techniques within a federated learning context, addressing system heterogeneity and domain shifts in a novel way.\n\n•\tquality: The framework is rigorously developed, with a clear methodology and implementation details provided. \n\n•\tclarity:This paper's figures are clear and easy to understand, and the explanations of the framework(MFP, DAR, Aggregation) are well-organized and coherent.", "weaknesses": "•\tThe introduction of the two issues, system heterogeneity and domain shift, in this article are not very clear. Regarding system heterogeneity, the author places emphasis on low-capability clients being unable to perform local training, but this emphasis lacks relevance to the rest of the paper. Additionally, the example of the cameras in the domain shifts problem is somewhat biased towards the \"multi-view task.\" For the issues of model heterogeneity and domain shift, the author needs to provide a clearer exposition of the application scenarios where these two problems intersect. If feasible, it would be beneficial to supplement the explanation with illustrations.\n\n•\tThe strategy employed in the Model Fusion Pruning phase is not particularly innovative. Moreover, this strategy requires the storage of the entire global model and additional extensive computations under resource-constrained settings, which severely contradicts the problem that this paper aims to address. Furthermore, the pruning strategy employed in this article is the unstructured pruning strategy, so the actual contribution to the problem of system heterogeneity is limited.\n\n•\tAlthought the framework diagram is already very clear, it is suggested to include model symbols corresponding to the algorithm in order to provide a clearer explanation of the model's changes at each step.\n\n•\tThe method details in this article bear some resemblance to existing work, and there is a lack of references to these works. ( Diao, Enmao, Jie Ding, and Vahid Tarokh. \"Heterofl: Computation and communication efficient federated learning for heterogeneous clients.\" arXiv preprint arXiv:2010.01264 (2020), Yuang Jiang, Shiqiang Wang, Bong Jun Ko, Wei-Han Lee, and Leandros Tassiulas. 2019. Model Pruning Enables Efficient Federated Learning on Edge Devices. CoRR abs/1909.12326 (2019). )\n\n•\tIn addition, there are some approaches that address model heterogeneity and domain shift problems through distillation. Although this paper focuses on pruning paths, these methods should also be mentioned and even compared. (Zhu, Zhuangdi, Junyuan Hong, and Jiayu Zhou. \"Data-free knowledge distillation for heterogeneous federated learning.\" International conference on machine learning. PMLR, 2021. Huang, Wenke, Mang Ye, and Bo Du. \"Learn from others and be yourself in heterogeneous federated learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. Below we address your concerns in order.\n\n\n## Response to Weakness 1:\n\n**Clarification on System Heterogeneity:**\nIn our paper, we emphasize that low-capability clients \"fail to complete\" local training rather than being \"unable to perform\" local training. Following the conventional setting [1,2,3], low-capability clients (i.e., stragglers) possess the basic capability to train models. However, they fail to complete training within the maximum allowed time due to limited resources, such as battery status or CPU power. MFP addresses this issue by reducing resource consumption, thereby enabling the stragglers to participate in FL.\n\n**Clarification on Domain Shifts Example:**\nImages collected by the cameras are not exclusively \"multi-view\" data, particularly when factors such as lighting conditions or weather vary which are classified as domain shifts. Additionally, images with different view angles only (i.e., \"multi-view\") are also multi-domain data. For instance, previous work [4] has utilized rotated images to create multi-domain datasets.\n\n**Application Scenarios and Illustration:**\nThank you for your insightful comments. Due to the length limitations, please refer to **Responses to Weakness 3 \\& Question 2** from **Reviewer Ct3S** for the details.\n\n>[1] Lim, Wei Yang Bryan, et al. Federated learning in mobile edge networks: A comprehensive survey. IEEE Commun. Surv. Tutorials 2020.\n>\n>[2] Nguyen, Dinh C., et al. Federated learning for Internet of Things: A comprehensive survey. IEEE Commun. Surv. Tutorials 2021.\n>\n>[3] M. Gerla, et al. Internet of Vehicles: From intelligent grid to autonomous cars and vehicular clouds, WF-IoT 2014.\n>\n>[4] Nguyen, A. Tuan, et al. FedSR: A simple and effective domain generalization method for federated learning. NIPS 2022.\n\n\n## Response to Weakness 2 \\& Question 1 \\& Question 2:\n\n**Innovative and Practical Differences of MFP:**\nAs detailed in lines 143-160, Algorithm 1, and Fig. 1, a dynamic model fusion mechanism is proposed within MFP. This mechanism is distinct from regular FL. Inspired by transfer learning, DapperFL incorporates the global model into the fine-tuned model to assimilate cross-domain knowledge (formally represented in Eq. 1). Furthermore, a dynamic adjustment mechanism controls the quality of transferred knowledge (as described in Eq. 2), preventing overfitting to the local domain and improving the model's generality. \nOur experiments in the presence of domain shifts show that DapperFL with MFP improves accuracy by up to 7.45\\% (on the Digits benchmark) compared to SOTA frameworks that support system heterogeneity.\n\n**Clarification on Additional Computations and Global Model Deployment:**\nIt is important to note that MFP does **NOT** entail additional computations. MFP requires clients to fine-tune the model during the initial epoch, consuming resources comparable to regular FL. In subsequent epochs, the pruned model undergoes updates, significantly reducing resource consumption (up to 80\\%). Moreover, fine-tuning the global model for at least one epoch is essential in popular FL methods to capture data features, and this is not unique to ours.\n\n**Clarification on Pruning Strategy:** \nWe would like to highlight that, the pruning strategy used in DapperFL is structural pruning (channel pruning), rather than unstructural pruning. \nChannel pruning is hardware-friendly and can be easily implemented with popular ML libraries such as PyTorch, making it suitable for edge devices. \nWe provide our code in the Abstract, which details the implementation of our approach.\nWe want to emphasize that \"unstructured pruning\" is not presented in our paper. Nevertheless, we acknowledge the importance of clarity and will include a description of the adopted structural pruning.\n\n\n## Response to Weakness 3:\n\nWe appreciate your constructive suggestion. Specifically, the global model transitions to $\\mathcal{W}^t$, the fine-tuned model transitions to $\\boldsymbol{w}^{t,i}\\_{ft}$, and the fused model transitions to $\\boldsymbol{w}\\_{fs}^{t,i}$.\n\n\n## Response to Weakness 4:\n\nThank you for highlighting these studies that offer valuable insights into heterogeneous FL and model pruning. We will incorporate them into our paper to discuss their relationship to ours.\nDespite the similarities, our work contributes novel solutions by focusing on domain shifts and the integration of MFP, DAR, and heterogeneous aggregation. \n\n\n## Response to Weakness 5:\n\nThank you for highlighting these important works.\nWe did not include distillation-related work as they typically require maintaining both a teacher model and a student model. This imposes a significant burden and is impractical in our targeted scenarios. \nNonetheless, we acknowledge the relevance of them and will incorporate them to enrich our literature review.\n\n\n## Response to Limitation 1:\n\n**Hyper-Parameter Selection:**\nPlease refer to **Responses to Weakness 1 & Questions 1** from **Reviewer 1ipw** for the details.\n\n**Solution for System Heterogeneity:**\nPlease refer to **Response to Weakness 2 \\& Question 1 \\& Question 2** for the details.\n\n**Theoretical Convergence Analysis:**\nIt is noteworthy that extensive prior work has established the convergence of model pruning-based FL [5,6,7,8]. Our DapperFL, which also incorporates model pruning, does not violate the convergence of FL. The experimental results in our paper further support the convergence of DapperFL.\n\n>[5] Zhou, Hanhan, et al. On the convergence of heterogeneous federated learning with arbitrary adaptive online model pruning. arXiv 2022.\n>\n>[6] Li, A., et al. Hermes: an efficient federated learning framework for heterogeneous mobile clients. MobiCom 2021.\n>\n>[7] Zhida Jiang, et al. Fedmp: Federated learning through adaptive model pruning in heterogeneous edge computing. ICDE 2022.\n>\n>[8] Jiang, Yuang, et al. Model pruning enables efficient federated learning on edge devices. TNNLS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their time and appreciate all the detailed reviews and constructive feedback. Extra experimental results and illustrations are presented in the rebuttal PDF file to address some common concerns raised by reviewers. In addition, each review will be replied to individually. The discussion here will be properly incorporated into a new version of our paper.

Meta-review: The paper proposes a novel application scenario. The paper is well written. The unclear parts are resolved by rebuttal. All reviewers agreed to accept this paper.
Decision: Accept (oral)