FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion
Authors: Zhenheng Tang
Yonggang Zhang
Peijie Dong
Yiu-ming Cheung
Amelie Chi Zhou
Bo Han
Xiaowen Chu
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Federated Learning
communication efficiency
causality
Primary area: other
TL;DR: This work identifies the cause of low performance of one-shot FL, and proposes FuseFL to progressively train and fuses DNN model following a bottom-up manner, reducing communication costs to an extremely low degree.

Abstract: One-shot Federated Learning (OFL) significantly reduces communication costs in FL by aggregating trained models only once. However, the performance of advanced OFL methods is far behind the normal FL. In this work, we provide a causal view to find that this performance drop of OFL methods comes from the isolation problem, which means that local isolatedly trained models in OFL may easily fit to spurious correlations due to the data heterogeneity. From the causal perspective, we observe that the spurious fitting can be alleviated by augmenting intermediate features from other clients. Built upon our observation, we propose a novel learning approach to endow OFL with superb performance and low communication and storage costs, termed as FuseFL. Specifically, FuseFL decomposes neural networks into several blocks, and progressively trains and fuses each block following a bottom-up manner for feature augmentation, introducing no additional communication costs. Comprehensive experiments demonstrate that FuseFL outperforms existing OFL and ensemble FL by a significant margin. We conduct comprehensive experiments to show that FuseFL supports high scalability of clients, heterogeneous model training, and low memory costs. Our work is the first attempt using causality to analyze and alleviate data heterogeneity of OFL.

FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion
Zhenheng Tang"* Yonggang Zhang! Peijie Dong
Yiu-ming Cheung
Amelie Chi Zhou' Bo Han Xiaowen Chu
t Department of Computer Science, Hong Kong Baptist University
: DSA Thrust, The Hong Kong University of Science and Technology (Guangzhou) {zhtang, ygzhang, ymc, amelieczhou, chanm1]@comp .kbu.. edu. .k {pdong212, xwchu)@connect. httpiconnnct...uut....... edu. cn
Abstract
One-shot Federated Learning (OFL) significantly reduces communication costs in FL by aggregating trained models only once. However, the performance of advanced OFL methods is far behind the normal FL. In this work, we provide a causal view to find that this performance drop of OFL methods comes from the isolation problem, which means that locally isolatedly trained models in OFL, may easily fit to spurious correlations due to data heterogeneity. From the causal perspective, we observe that the spurious fitting can be alleviated by augmenting intermediate features from other clients. Built upon our observation, we pro- pose novel learning approach to endow OFL with superb performance and low communication and storage costs, termed as FuseFL. Specifically, FuseFL decom- poses neural networks into several blocks and progressively trains and fuses each block following bottom-up manner for feature augmentation, introducing no additional communication costs. Comprehensive experiments demonstrate that FuseFL outperforms existing OFL and ensemble FL by significant margin. We conduct comprehensive experiments to show that FuseFL supports high scalability of clients, heterogeneous model training, and low memory costs. Our work is the first attempt using causality to analyze and alleviate data heterogeneity of OFL
Introduction
has become a popular paradigm that enables collaborative model training without sharing private datasets from clients. Two typical characteristics of FL limit its performance: (1) FL. normally has non-lIl (Independently and Identically Distributed) data, also
Federated learning (FL) 966 67
called data heterogenetity [67 which causes unstable slow convergence [68 146 124
135 and
poor model performance [177 94
134 [50); (2) The extremely low bandwidth, e.g. ~~ 10 MB/s of FL in Internet environments 66
1B3 129 128 132 129 leads to high communication
time of large neural network. For example, communicating once ResNet-50 [5]] with 25.56M
parameters (102.24MB) or GPT-3 [4] with 175B parameters (700GB) will consume around 102.24 24 seconds or 194 hours, respectively. Current FL methods alleviate this problem by skipping the gradient synchronization of traditional distributed training to save communication costs (7).6)). But the required hundreds or thousands of communication rounds still make the communication time unacceptable.
"Tii work is partially done during the visiting in The Hong Kong University of Science and Technology (Guangzhou).
Correspondence to Xiaowen Chu wewwhu@@hkssteee hrrichuu@hhusttg..dd..orr
"hhe code is publicly available:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To reduce the communication costs at extreme, one- FL (OFL) [59 38 81 164 25 23 only communicates the local trained model once. Thus, the communication cost the model size S for each client, less than FedAvg-style algorithms for hundreds or thousands of times. However, averaging for only once cannot guarantee the convergence of FedAvg. Thus, the direct idea is to aggregate client models on the server and conduct inference as ensemble learning does. Some advanced works also consider better model averaging [63 112 90 6], neurons matching [5]1 selective ensemble learning [26] [22 [43], model distillation 8 66 25 23 These methods may be impractical due to the requirements of additional datasets with privacy concerns, and the extra large storage or computation costs. Most importantly, there still exists large performance gap between OFL and the normal FL or the ensemble learning. This motivates the following question:
How to improve FL performance under extremely low communication costs with almost no extra computational and storage costs?
to analyze the performance drop of OFL. We firstly construct causal graph model the data generation process in FL, where the spurious features build up the data heterogeneity between clients, and invariant features of the same class remain constant each client (domain) 3 [T9 22
In this work, we provide a causal view [009] 110 3;
[5T [61]. Then, show the performance drop comes from the isolation problem, which means that locally isolatedly trained models in OFL may easily fit spurious correlations like adversarial shorteuts (40) 35] [33, instead of learning invariant features 33 119 causing a performance drop of OFL on the test dataset. Consider a real-world example, Alice takes photos of birds in the forests, while Bob near the sea. Now, the isolated models will mistakenly identify birds according to the forests or the sea [53] 351. Based on the causal graph, we intuitively and empirically show that such spurious fitting can be alleviated by augmenting intermediate features from other clients (Section 3
Built upon this observation, propose simple yet effective learning approach to realizing OFL with superb performance and extremely low communication and storage costs, termed as FuseFL, which builds up the global model through bottom-up training and fusion to improve OFL, performance (Section 4. Specifically, we split the whole model into multiple blocks ( (The "block" means single or some continuous layers in a DNN.). For each block, clients first train and share their local blocks with others; then, these trained local blocks are assembled together, and the features outputted from these blocks are fused together and fed into the next local blocks. This process is repeated until the whole model is trained. Through this bottom-up training-and and-fusiooooio method, local models can learn better feature extractors that learn more invariant features from other clients to avoid the isolation problem. To avoid the large storage cost, given the number of clients M, we assign each local client with small model with reduced hidden dimension with ratio v M, to ensure the final learned global model has the same size as the original model.
Our main contributions can be summarized as follows:
We provide a causal view to understand the gap between multi-round FL and OFL, showing that augmenting intermediate features from other clients contributes helps improve OFL. As far we know, this is the first work using causality to analyze the data heterogeneity of OFL.
To leverage causality to improve OFL, we design FuseFL, which decomposes models into several modules and transmits one module for feature augmentation at each communication round.
We conduct comprehensive experiments to show how FuseFL significantly promotes the perfor- mance of OFL without no additional communication and computation cost.
2 Preliminary
2.1 Federated Learning
In FL, a set of clients M = {m/m E 2,,,..., MM have their own dataset Dm- Given C' classes indexed by [C]. a sample in Dm is denoted by (z, E X' x [C], where x is the input in the space X' and is its corresponding label. These clients cooperatively learn a model F(0, x) X' + R that is
Note that this method is general to any neural network, and the fusion is different from averaging.

---

parameterized as 0 R.. Formally, the global optimization problem can be formulated as [96]
M
M
min L(0) A
Pm (0)
=
Σ m==
Pm Dm ((F;
m==1
where the local objective function of mtth-client 2F v) - CF(Ú. y) with & a F(0;2), CE denotes the cross-entropy loss, Pm > and E Pm 1. Usually, Pm 2 Rm/N, where nmm denotes the number of samples client (7.m (Dml) and N -Em= Pom.
The classic FL algorithm FedAvg (96). each communication round the central server randomly samples a part of clients S S M and broadcasts the model 0 to all selected clients, and then each m-th client performs multiple local updates. After local training, all selected clients send the optimized m to the server, and the server aggregates and averages local models to obtain a global model. Such multi-round communication introduces large communication costs [67]
2.2 Ensembled FL
The FedAvg requires multiple communication rounds for convergence 146 83
which might be
extremely large [67]. Given the model size S, FedAvg-style FL methods introduce communication costs as 7' x S. As shown in [able the current lowest communication cost of FL is reduced as S in OFL, making FL possibly deployable in low communication bandwidth scenarios [33 129]. Thus, we analyze what causes the performance drop of OFL and how to improve it. As the performance of average-baase based and model distillation OFL methods upper bounded by ensemble leamiigg [159 38 81,1644 23), we mainly focus analyzing ensemble learning and differentiating FuseFL from it. The output of the ensemble learning can be formalized as: (x) ) A N Emem 1i (0m; in which the local model Flle parameterized with 0m is isolatedly trained with minimizing empirical risk minimization (ERM) objective 152
22
function ((F(0, x). 3). (x,,) y) Dm, by SGD.
Federated Learning: A Causal View
3.1 The Sequential Structure of Neural Networks
neural network can decomposed into the sequential module-wise structure shown in Figure Formally, it can be defined as:
Higher possibility of fitting on RT R
possibility of fitting R7 or R
A -- for S K, (1) where is the final classifier, and Hg is the module that may consist of single multiple blocks. The A and each Hg are parameterized by @ E Rdd and 0* Rdd The H' H's (-) means H' (+))..)) Thus, the parameter 0 of F are con- catenated by the A and { E 1,2, ... K). and dA +> +E ==
f1
f2
f2
11
No
Vuggmented
H
H
H
H
Sartnnn Featuro
u1
B
Pilter Scatares
As Figure illustratts, each module H receives the output of module 1/-11 and the final classi- fier receives the output of the final hidden mod- ule HK and makes predictions δ = f() on the input æ. We call the output from each mod- ule, h = H (hhh11) and h = H (x), as the feature for simplicity.
R
gins
R
plav
Re
giev
(a) Isolated Training Ensemble
(b) Federated Fusion
Figure Structure Equation Model 109 of FL.
3.2 Structure Equation Model of FL
Inspired from the analysis of out-of-distribution
(OOD) generalization [3] through the lens of mutual information [12 2 and structure equation

---

model (SEM) in causality [110; [61]. we define the data generation SEM of FL as shown in Figure For local training dataset P,,. client m. the SEM is Y. + R + Xm R where Rey and Rm are invariant and spurious features, Ym and Xm are label and input data respectively. Here, the dataset D, is a subset of the whole dataset D. The Rig. is actually the nuisance at a global level (respect to Y), being independent of Y, but dependent on Xm
Non-IID data and causality. For a groundtruth label Y, its corresponding invariant features Rij do not change across clients B TT9 22, [ST]. However, the spurious features It are other factors that occasionally exist in data and do not have a relationship to Y. which means that the heterogeneous features of data (non-iid) with the same class come from the spurious features Rm (concept shift [67]). For example, in photos of birds in the forests or the sea, pixels of birds are Riny while the forests and the sea are Rm Considering the test dataset includes all client data distribution and even OOD data, the Yiest is largely dependent on - P(Yiea) X ((s)) ) >> P(Yies! [Xiea Ries) [
Non-IID scenarios. This SEM model considers the label shift (p:(y) # Py(W)) and concept shift (x:(y)y) * py(æly)) scenarios [67] [0], or both of them appear simultaneously. When the suppor I' of Ym is different or partly overlapped between clients m 1, .... M, this would be the severe non-IID scenario 155]. And it is obvious that spurious features It relate to the concept shift.
Spurious fitting. By conducting isolated local training on local dataset Dm, at client m, the model Fir is prone to learn to predict Ynm based on spurious features Rm i.c. low distance suu =
d(P(Y XXm, Rx P(F X R) but high distance d(P((,m |Xm: R ). P(F IXm. Rm )) in which the distance could be CE loss or KL divergence. The reason for the spurious fitting by isolated training is that the invariant features f from other clients are not observed by
client m, while the 1E frequently appears in the local dataset D,,, like the adversarial attacks or shortcuts [40] 35 [5]. This guarantees low error on the training dataset Dm, because it has much less data than Dess and D.....M, thus introducing high probability P(Ym||Xm. Rnn"). However, on test dataset Pist the low doe. of model Flee but high de of model F leads to high test error. Different from isolated training, FedAvg alleviates this problem by multiple times of averaging models to find those common features, including more R M and removing Rasp Ri.....
Feature augmentation. Through the above analysis, the key to improve OFL performance is to endow OFL with the ability of training to see invariant features across all clients. It has been found that training on noised datasets with SGD to optimize ERM can still result in some feature representations consisting of both spurious and invariant features; exploiting the invariant features is the key to helping improve OOD performance [58 3 g In light of this, we introduce augmenting features by fusing client models block by block. Concisely speaking, in FuseFL, each local model can conduct local training with the view from other clients Hipm (Xm), which helps filter out INP but retain Rinn as other clients cannot see R in their dataset Dism- This method can be seen as a kind of invariant feature augmentation 222 The details of FuseFL are shown in Section
3.3 Mutual Information
The goal of FL is to obtain a model that performs well on all client datasets 167 Thus, here we consider the random variable X, Y sampled from the global dataset D. In this section, we also write H as the features that output from (H ... (H² (X))) for simplicity.
Given the probabilistic graph model (Regu YY + X - H¹ - > H - F(X) (Eq. D. where Rinh are ignored for simplicity, the MI between Y and subsequent transformations H² on X satisfies a decreasing trend: I(X;Y) > Y(H > ... 2 1(H*; the MI between X and subsequent transformations on X satisfies a decreasing trend: Entropy (X) 2 x 2 > I IH :X [122. If I(H*; RTPe) = O and H² can predict labels, H² is called invariant features so that the final classifier will not overfit to spurious correlations between RST and Y. The
previous works [122 show that achieving the following minimal sufficient statistic provides good
'Without loss of generality, one can also model the SEM of test data as Yest + Rim + Xiest for simplicity. 'Support is the region where the probability density for continuous random variables (probability mass function for discrete random variables) is positive B]

---

generalization:
Sufficient statistic: I(X;Y) (H(()))))
(2)
Minimal statistic: H(X) arg min I XX);
((X)
Lemma 3.1 (Invariance and minimality (2.. Given spurious feature R for the label Y, and probabilistic graph model (R"" Y) X (X), then,
[XH(;); RTTh) S (X;;;;; x) I(X;Y).
There is nuisance R such that equality holds up residual
(XH;;;; RT (XH()): X) I(X;Y) -
where - ((H(X): R I(X; Y). The sufficient statistic H (X) (satisfying Eq. is invariant Ryw and only if is minimal (satisfying Eg. 3
Remark 3.1. Based on Lemma 3.1 we can study how (X);; and I((H); Y) changes to study what degree the (X) contains spurious features.
5..5
0.50
0.60
Local 21
Load 33
Tualsa 21
Extimttoon NY.
0.90
Centraliaed
Training
85
80
75
0.85
70
0.80
0.35
80
0.75
0.25
50
0.20
0.70
Module Index
Module Index
Module Indee
(a) Estimated MI I(H*; ; X).
(b) Estimated MI I(HI; Y).
(c) The separability of layers.
Figure 2: Estimated MI and separability of trained models with non-IID datasets.
0.90
80
0.80
Text Accurary
70
0.34
0.70
60
- Tuan (1
0.65
Lecd Tuln
0.60
50
0.55
Module Index
Module Index
Module Index
(a) Estimated MI T(TT*; X).
(b) Estimated MI T(TT Y)-
(c) The separability of layers.
Figure 3: Estimated MI and separability of trained models with non-lID backdoored datasets.
Empirical study. We empirically estimate the T(H X) and (H Y) of isolated local trained features, and the T(H X) and T(H Y) of augmented features. the deep neural networks (DNNs) show layer-wise feature enhancements 7 we also measure the linear separability [7] [01 of features H& and H to see how they change. Details of MI estimation and linear separability shown in Appendix D.3 and D.4| The experiments are conducted by training ResNet-18 18 with CIFAR-10 74 partitioned across M = clients. Figure 2 shows the local features H/ have significantly higher THH* X) lower T(H* Y) than augmented features H With the increased non-IlID degree (lower a), the I [H Y) decreased further, demonstrating that the local feature Hi fits on more anti-causal relationship between senu and Y... The fused high-level features show better linear separability. And H.&. is more robust to Iep
Except for the natural spurious features that exist in CIFAR-10, also study the effect of spurious features by handcraft. Specifically, we inject backdoored data samples [10] 997 out of clients as Dhack in which the images have handcrafted textures generated according to the labels as strong anti-causal relation. Details of backdoored datasets are introduced in Appendix [D.5] Figure shows that the backdoor features lead information loss in X. And the backdoored data samples further aggravate the information loss of label Y. The isolated local trained features retain significantly less [HH, Y) than augmented features.
5

---

Training
Communicated shared across Clients
Data flow local clieuts Data flow on other clients
Modules on difference clients
Module
Adapter
Dj
Dataset
f1
fz
fa
f1
f2
f
f3
Aa
Aa
A2
11
H3
H1
H5
H
1
m
H
H
H
H
H
H
H
H3
11
D2
22
D2
D2
D:
D2
D2
Diret
(a) Isolated Training
(b) Fusing L1
Training
(c) Fusing L2 & Training
(d) Deployment
Figure 4: (a) Initially, all layers are isolated training. Note that the layer here does not only mean one or Conv layer, but generally refers to neural network block that can consist multiple layers. (b) Then, all first blocks (I.1) of different clients are communicated, shared and frozen among clients. Then, the adaptors are added behind the fused block, to fuse features outputted from the concatenated local blocks. (c) Train the third blocks (L3) follow the similar process in (b). (d) inference process of FuseFL. The larger squares represent the original training block in local models. The smaller squares adaptors that fuse features from previous modules together, which are 1 Conv kernels or simple average operations with little or no memory costs. Note that (a) also represents local training in ensemble FL, where different clients train models on local datasets.
4 FuseFL: Progressive FL Model Fusion
Motivated by analysis in Section we propose augmenting local intermediate features H on client m, which helps reduce fitting to a spurious correlation between Y, and RTTh However, direct fusing features together to make predictions faces the following problems.
Altered feature distribution. During local training, the local subsequent model r A Am H Hk+1 after block & on client m is trained based on local features H After feature fusion, changed local features lead to feature drifts [75 134
m
Mismatched semantics. Each local feature H has totally different distributions, scales, dimensions; thus, directly averaging may cause useful features to be overwhelmed or confused by noisy features.
4.1 Train, Fuse, Freeze, and Re-Train
As Figure And Algorithm D (Appendix TD show, the main loop of FuseFL including training, fusing, and freezing, which is repeated for all K split blocks following progressive manner. Note that for H1 there no layer fusion, which is isolatedly trained.
Fuse. After each local training step, clients share their H with other clients, and fuse them following Eq. An adaptor stitched before Hk+ (Section 4.2 Then the local model becomes as Fis following Eq. 55]
Freeze and re-train. To address the altered feature distribution problem (A (1:) - H (x)), for each step l client m, the subsequent layers k will be trained again based on D... Thus, the rk+1 can learn from all low-level features of all clients. Note that we do not need to train each block & with the same epochs isolated training, because the DNNs naturally follow the layer-wise convergence [13 48

---

H (2) HH (z), .... H()(z))
(4)
Fik. Am o H .... o Hnt 1 o H fis o .... o H
(5)
tus
Fius A His H o... Hin.
(6)
A Awg- Average His (a))-
Aconv Conv(Coneat(HF His (x))).
(8)
By freezing fused blocks and retraining high-level models, the another benefit is to enforce the SGD to use previous features from other clients to continue tuning the high-level model. The previous local trained high-level models may overfit on shortcut features from the noisy data. This insight also utilized in defensing adversarial attacks [07]
4.2 Feature Adaptation
To address the mismatched semantics problem, the intuitive approach is to preserve the original feature structures through the concatenation of all features to the next block. However, this leads to problems: (1) requiring modification of subsequent modules; (2) feature size explosion of subsequent blocks by O(M*). To address these two problems, we introduce an adaptor stitched before local modules (k > 1), and training together with r As an initial trial to operationalize FuseFL, we utilize conv x as the adapter as Eq. 8 We also verify the use of average as adapter (Eq. in experiments (Section 6).
4.3 Benefits of FuseFL Design
Mitigating fitting on spurious correlations. During the local training on datasets with spurious features, the final learned representations with ERM still contain some invariant features 3 22] Thus, some work proposes to finetune the classifier based on data samples with invariant features to let the classifier make predictions based on invariant features 770 61 [06]. Similar to this motivation, we hope to incorporate other client modules as auxiliary feature extractors to generate more invariant features of local data during training subsequent layers. Figure describes the mechanism that using other local models Hism (Xmm) (Xm) help to filter out spurious features Rt, but retain R As other clients (i o m) cannot see Rit in their dataset Diym during local training, only invariant features can pass through Hi (Xm). This method can be seen as the invariant feature augmentation [22]
Saving storage and communication costs than ensemble FL. Similar to ensemble learning, directly collecting and fusing local models together will enlarge the total model size from S to S x M. Note that FuseFL actually builds up global model with blocks fused together, with the hidden dimensions (channels) enlarged from n, n, M. Thus, intuitively, we reduce the hidden dimension of the local model n to reduce the memory requirements. Interestingly, with a scaling ratio ; when scaling all local linear convolutional layers, each matrix should be scaled on both input and output dimension as ny = "Mng. The ratio of memory costs between FuseFL and the original single model is Im = M x n (2,,,,, To obtain r. = 1, we obtain the scaling ratio = VM, which means that FuseFL can keep similar memory requirements with the original model size with reducing hidden dimensions as ratio VM, demonstrating good theoretical scalability to the M. We will verify this in experiments (Section 6.2
Privacy concerns. FuseFL only shares layers between clients, which aligns with other classic and advanced FL methods in all directions mentioned in Section 5
Support of heterogeneous models. The block in FuseFL does not mean single linear convolution layer, but a general module that can consist of any DNN, thus supporting FL with heterogencous models (see experiments 6.2). The adaptor can be designed to transform features of different shapes to align with the input of the next local block.
Layer-wise training to reduce training epochs. Because each communication round means multiple local training epochs. To keep the total training epochs the same as the one-shot FedAVG (represented as E), we assign the local training epochs of the FuseFL as E/ K. Thus, the number of total training epochs FuseFl. is the same as other OFI. methods. The core insight of this design can be referred to as the progressive freezing during training DNNs [113].
7

---

5 Related Works
5.1 Data Heterogeneity in FL
The notorious non-lID data distribution in FL severely harms the convergence rate and model performance of FL. The model regularization proposes to add penalty of distances between local and global models [177 [] Feature cal- ibration aligns feature representations of differ- ent clients in similar spaces (29, 134 60, 135].
Table I: Demystifying different FL algorithms. T represents communication rounds, S the model size, M the number of clients. The "Centralized" means training the model with all datasets aggre- gated with SGD. "Comm." means communication.
FedMA 141 exploits layer- wise communica- tion and averaging methods, in which the aggre- gation is conducted on fine- fine-grained layer. Thus, its linear dependence of computation and com- munication the network's depth, is not suit- able for deeper models, which introduces large computation costs ve-traiming and more com-
Method
Coenm. cust
Sterage and
Support model heierogemeity
Not require
estra data
FedAvg
8
Avcragoshased OFI,
Ensemble bused
Model distillation UFL,
PuseFi
S
munication rounds 19 58]. Unlike FedMA, FuseFL introduces block-wise communication and aggregation with much less communication rounds and computation costs. Furthermore, due the matching and averaging aggregation, FedMA only supports linear or Conv layers, which severely limits its practical usage. By viewing the separated block as black box and concatenating output features, FuseFL can successfully support merging any kind of neural layer. Some works on fairness analysis in FL also relate to this work in perspective of local and global characteristics [42 32
5.2 One-shot FL
One-shot FL [199 38 81 164; 25 reduces communication costs from T x S to S by communicating with only one round, Average-based methods focus on better averaging client models, like Fisher
information 63 [22,, bayesian optimization [90] 6 matching neurons [5] 141]. However, the non- linear structure of DNNs makes difficult obtain a comparable global model through averaging. Ensemble-bard based methods make prediction based on all or selected client models 26 52, 143], but
requires additional datasets with privacy concerns. And they have low scalability of the number of clients due to the storage of client models. Model distillation uses the public [81] or synthesized datasets [64 255 to distill a new model based on ensemble models [38 8T. These methods may be impractical in data-sensitive scenarios, such as medical and education, or continuous learning scenarios 28 27 Furthermore, there exists a large performance gap between these methods and the ensemble learning.
Due to the limited space, we leave the detailed reviews in Table X and Appendix C Table T concisely demystifies different FI. methods in terms of communication cost, storage cost, model performance, and whether supporting model heterogeneity or requiring extra data.
6 Experiments
6.1 Experiment Setup
Federated Datasets and Models. In order to validate the efficacy of FuseFL, we conduct compre- hensive experiments with commonly used datasets in FL, including MNIST [77], CIFAR-10 (73). FMNIST (47), SVHN [100], CIFAR- 100 733 and Timy-Imagenet (76). For studying the non-IID problem in FL, we partitioned the datasets through a widely-used non- partition method, namely Latent Dirichlet Sampling (56) 67 4 80), in which the coefficient a represents the non-lID degree. Lower a generates more non-IID IID datasets, and vice versa. Consistent with established practices in divided with three distinct degrees of non-IllD with a E {0.1, 0.3.0.33000 0.5}. there is no additional explanation, the non-lID degree a is set to 0.5 by default.
the field 159 1 114, 93) 80), each dataset
Following other classic and advanced FL works studying non-llI) problems and communication- efficient FL. [25 52 93. 159], we train R8 [51) on all datasets in main experiments. And we reduce and increase the number of layers ResNet- 10 and ResNet-26 to verify the effect of FuseFL
8

---

in model- hettreeeeerggeeettty FL. The number of clients is set as M = 5 by default. Moreover, we study
the scalability of our methods with different M E {5. 10, 20, 50}.
We use SGD optimizer with momentum coefficient as 0.9, and the batch size is 128. The number of
local training epochs F 200. We search learning rates in {0.0001, 0.001, 0.01, 0.1] and report the
best results. The detailed hyper-parameters of different settings are shown in Table of Appendix D
Table 2: Accuracy of different methods across a = {0.1, 0.3, 0.5) on different datasets. Ensemble means ensemble learning with local trained models, which is upper bound of all previous methods but impractical in FL due to the large memory costs and the weak scalability of clients. Thus, we highlight the best results in bold font except Ensemble.
Dotases
MINIST
FMNIST
CIEAR 10
SVHN
CIEAR 100
Tiny Imagenet
Method
a-40.1
cw40.5
1-0011
cH40.3
--0.
a-0011
ow0.5
[H-0.2
c-003
0-40.5
[vw0.1
co00.3
cu40.5
-=0,1
(24033
ci-0.5
RedAvg
48:24
72.54
511.5
K2.96
55.72
23:95
22:22
03.87
61.51
26.19
4.38
11.51
12:11
5.12
10:46
11:89
FedDiF
74.01
50:67
40:58
46.78
53.56
49.13
73.98
18.17
30.38
36.35
18.23
27.43
75.03
53.45
15:33
90.21
DENSE
76.48
55.54
55.34
79.59
42.07
32.34
Ensenbble
$6.81
96.76
97.22
67.71
57.25
89.42
57.5
77.35
79.91
65:29
85.31
85.7
35.69
09.41
53.39
30.85
39.43
45.5
FuroFI K =
97422
96.43
98.54
M5.15
19.94
89.57
70.85
$1.41
64.54
91.07
99.87
36.117
05.12
46 12
29.28
11:11
34.54
FuseFL K
97.19
08.3 34
98.3 29.29
83.00
$4.55
90.50
73.79
$4.58
81.15
78.08
89.63
69.34
36.86
42.70
49.30
27.63
33,04
34.28
FuseFL K
96.65
98.35
98.15
83.3
$8.57
88.24
70:46
90.70
74.99
50.31
38.58
$9.94
34.97
39.08
40.73
25.21
32.59
33.82
Baselines. Except the classic baseline FedAvg 196] and advanced OFL method DENSE [159] ap-
ply two prevailing data-free KD methods DAFL 18] and ADI [153] into OFL. We choose FedDF [88] as its high efficiency in few-round FL. We conduct ensemble FL as it is the upper bound across ensemble-and -aaembllaanddddiilllttionn based methods, yet impractical in real-world scenarios. The communication round for all baseline methods is only 1. For our method FuseFL, the number of communication rounds is equal to the number of splitted blocks K. However, the actual communication cost is as same as one-shot FL. Because FuseFL only communicates part of the model. After all rounds, the total communication cost is SM, where S is the model size, M the number of clients.
6.2 Experimental Results
Main Results. Table 2 shows that FuseFL generally outperforms all other baselines except for ensemble FL. All ensemble-and-distilaatin baselines have lower performance than ensemble FL. Nevertheless, by the insights from causality (Section 3 and our innovative design (Section 4.. FuseFL can significantly outperform ensemble FL for almost all cases except for CIFAR- 100 with = 0.3, 0.5 and Imagenet We suppose the reason is that CIFAR- and Tiny- Imagenet has much more data divergence between different classes, thus the overlap between Rij is much less than other datasets. Recall that the benefits of FuseFL. come from fusing sub models training on other clients, thus filtering out 1e and collecting Rijn of the same class to improve the generalization performance. The large data divergence in C1FAR-100 and Tiny-Imagenet limits benefits of FuseFL.
Table 3: Accuracy with FuseFL with Table 4: Memory Occupation. For different number conv 1x 1 or averaging to support heteroge-of clients, the number of basic channels in ResNet-
neous model design on CIFAR-10.
18 of FuseFL is set as 32, 20, 14, 9 with M E {5, 10, 20, 50). respectively. Other OFLs refer to Fe- dAvg, FedDF, DAFL, Fed-ADI, DENSE
non-Ill) degree
a 0.1
a = 0.3
a= 0.5
Ensemble
57.5
77.35
79.91
FuseFL
73.79
84.58
81.15
# Clients
=
M 10
M 20
M 50
FuseFL (Avg)
68.08
71.49
80.35
Single Model
42.66MB
42.66MB
42.66MB
42.66MB
FuseFL-Hetero
75.33
81.71
82.71
OFLs
42.66MBB
42.66MB
42.66MBB
42.66MB
PuseFL (Avg)-Hetero
68.31
76.27
79.74
Ensemble
213.31MB
426.62MB
853,24MB
1033.10MB
FuseFL K
53.71MB
42.32MBB
42.13MBB
45.48MB
FusaFL K 4
55.38MB
44.92MB
47.22MB
58.63MB
FuseFL K
68
68MB
64.77MB
16.1
FuseFl (Avg)
53,32MB
41.66MB
4088MB
4MMB
Support of heterogeneous model design. Table 3 shows training heterogeneous model using FuseFL. In all M = 5 clients, 2 clients train ResNet- TO and other 2 clients train ResNet-26, the left client trains ResNet- 8. We set K 4 for FuseFL. Results show that training with heterogeneous models has similar or even better results than homogeneous models, demonstrating that FuseFL supports training heterogeneous model well. This will be very useful in heterogeneous computation environments.

---

Table 5: Accuracy of different methods across M
{5, 10. 20, 50)..
Dataset
CIFAR-10
SVHN
Method
M =
10
M = 20
50
10
M 20
50
FedAvg
43.67
38
36.03
23.01
56.09
45.34
47.79
36.53
FedDF
53.56
54.44
43.15
29.52
73.98
62.12
60.45
51.44
Fed-DAFL
55.46
56.34
45.98
29.41
78.03
63.34
62.19
54.23
Fed-ADI
58.59
57.13
46.45
27.45
78.85
65.45
63.98
57.35
DENSE
62.19
61.42
52.71
28.51
80.03
67.57
66.42
59.27
Ensemble
79.91
77.25
59.69
55.63
85.7
73.45
68.76
54,96
FuseFL. K 2
84.34
73.71
62.85
42.18
90.87
88.52
85.18
72.25
FuseFL, 81.15
78.28
62.57
37.08
89.34
89.31
86,94
45.49
FuseFL. K 8
74.99
67.35
63.19
28.28
72.65
64.11
42.19
Memory occupation. Table shows the mem- ory occupation with varying number clients and split modules. Results show that FuseFL requires similar memory with the single model when K 2, while the ensemble FL requires the x M storage cost. And using averaging feature fusion method can further reduce the memory cost. Table 12 shows that the FuseFL with K = 2 FuseFL L with averaging has parable or better performance than other variants of FuseFL.
Table 6: Local and global accuracy of 5 local client models. BDo and BD1 represent two clients trained on backdoored datasets. Normalo, Normal,, and Normaly represent three clients trained clean datasets.
Client
BDg
BD;
Normaly
Normal,
Normaly
Local Ace.
100.0
100.0
99.7
99.9
100.0
Global Acc.
32.6
27.1
38.4
Scalability the number of clients. We em- pirically prove that the memory occupation in- creases little along with the increased M (Ta- ble 4). and keeping performance outperforms than baselines (T.
Table 7: Comparing accuracy on backdoored CIFAR-10.
Influence on local models of backdoored datasets. As shown in Section 3. the back- doored features lead to information loss in X. Here we further show how the FL performance is influenced by the backdoored features. Ta- shows that the backdoored (BD) clients fit on the handcrafted spurious features, thus hav- ing lower global accuracy than normal clients.
# Backdoored clients
Maa =
Non-IID degree
0.1
0.3
0.5
Ensemble
52.76
74.88
73.46
FuseFL K = 2
43.99
75.61
84.79
FuseFL K = 4
55.99
78.39
80.52
FuseFL K = 8
45.66
74.01
82.12
Backdoored clients
Maa 2
Non-IID degree
0.1
0.3
0.5
Test accuracy on backdoored datasets. Table provides the accuracy of different methods training with backdoored CIFAR-10 10. The test dataset is the original clean test set. We test different numbers of backdoored clients Maa- 2 out of a total of 5 clients, to see how the degree of the backdoor influences training. Results show FuseFL outperforms ensemble FL in all cases. demonstrating that FuseFL can defend better against the backdoor samples than ensemble FL.
Ensemble
52.67
68.42
75.83
FuseFL K = 2
54.24
71.49
82.61
FuseFL K = 4
54.77
67.53
72.99
FuseFL K = 8
51.70
68.31
76.95
7 Conclusion
In this work, we draw inspiration from the causality and the information bottleneck to analyze the cause of low performance of ensemble FL and OFL. Specifically, the local isolatedly trained models are easily fit spurious features, local clients cannot learn more invariant features and remove spurious features from other datasets. Built upon this insight, we provide a novel approach FuseFL to augment features by fusing client layers a bottom-up manner, thus mitigating the spurious fitting and encourage learning of invariant features. FuseFL achieves OFL with extremely low communication costs with significantly higher performance than current OFL and ensemble FI. methods.
10

---

Acknowledgments and Disclosure of Funding
This work was partially supported by National Natural Science Foundation of China under Grant No. 62272122, the Guangzhou Municipal Joint Funding Project with Universities and Enterprises under Grant No. 2024A0310616,, the Hong Kong RIF grant under Grant No. R6021-20, and Hong Kong CRF grants under Grant No. C2004-21G 21 and C7004-22G. BH was supported by NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation Nos. 2022A151500 1652 and 2024A151500 2399, HKBU Faculty Niche Research Areas No. RC-FNRA-IGG/22-2SSCCI
and HKBU CSD Departmental Incentive Scheme. YGZ and YMC were supported part by the NSFC Rescarch Grants Council (RGC) Joint Research Scheme under the grant: 4-HKBU214/211 the General Research Fund of RGC under the grants: 12201321, 12202622, 12201323, the RGC Senior Research Fellow Scheme under the grant: BRFS2324-2S02, and the Initiation Grant for Faculty Niche Research Areas of Hong Kong Baptist University under the grant: RC-FNRA-IGG23-24SSI122
References
[1] D. A. E. Acar, Y. Zhao, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama. Feder- ated learning based dynamic regularization. In International Conference Learning Representations, 2021.
[2] A. Achille and S. Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 2018.
[3] K. Abuja, E. Caballero, D. Zhang, J.-C. Bignon-AAdet, Y. Bengio, I. Mitliagkas, and Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. Ad- vances in Neural Information Processing Systems, 34:343--3450, 2021.
[4] K. Ahuja, J. Wang, A. Dhurandhar, K. Shanmugam, and K. R. Varshney. Empirical or invariant risk minimization? sample complexity perspective. In International Conference on Learning Representations, 2020.
[5] S. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2022.
[6] M. Al-Shedivat, J. Gillenwater, E. Xing, and A. Rostamizadeh. Federated learning via posterior averaging: A new perspective and practical algorithms. In International Conference on Learning Representations, 2020.
[7] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
[8] M. Arjovsky, L. Bottou, I. Guirajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint 1907::9997.0283 2019.
[9] S. Babakniya, S. Kundu, S. Prakash, Y. Niu, and S. Avestimehr. Federated sparse training: Lottery aware model compression for resource constrained edge. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022), 2022.
[10] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938-2948, 2020.
[11] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D) Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018.
[12] S. Bibikar, H. Vikalo, Z. Wang, and X. Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6080-6088, 2022.
[13] I. Bistritz, A. Mann, and N. Bambos. Distributed distillation for on-device learning. Advances in Neural Information Processing Systems, 14:22593-22605 2020.

---

[14] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan. P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[15] K. Cai, X. Lei, J. Wei, and X. Xiao. Data synthesis via differentially private markov random fields. Proc. VLDB Endow., 14(11):2190-25 2200222222222 jul 2021.
[16] H. Chang, V. Shejwalkar, R. Shokri, and A. Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv: htt::/912:.0 1912. 2019.
[17] A. Chatalic, V. Schellekens, F. Houssiau, Y. A. de Montjoye, L. Jacques, and R. Gribonval. Compressive learning with privacy guarantees. Information and Inference: A Journal of the IMA, 05 2021. iaab005.
[18] H. Chen, Y. Wang. C. Xu, Z. Yang, C. Liu, B. Shi, C. Xu, C. Xu, and Q. Tian. Data-free learning of student networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3514-3522, 2019.
[19] H.-Y. Chen and W.-L. Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. In NeurIPS, 2020.
[20] H.-Y. Chen and W.-L. Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2021.
[21] T. Chen, Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[22] Y. Chen, W. Huang, K. Zhou, Y. Bian, B. Han, and J. Cheng. Understanding and improving feature learning for out-of-distribution generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[23] R. Dai, Zhang, A. Li, T. Liu, Yang, and B. Han. Enhancing one-shot federated learning through data and ensemble co- co-boosting. In The Twelfth International Conference Learning Representations, 2024.
[24] Y. Dai, Z. Chen, J. Li, S. Heinecke, L. Sun, and R. Xu. Tackling data heterogeneity in federated learning with class prototypes. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[25] D. K. Dennis, T. Li, and V. Smith. Heterogeneity for the win: One-shot federated clustering. In International Conference on Machine Learning, 2021.
[26] Y. Diao, Q. Li, and B. He. Towards addressing label skews in one-shot federated learning. In The Eleventh International Conference on Learning Representations, 2023.
[27] J. Dong, H. Li, Cong, Sun, Y. Zhang, and L. Van Gool. No one left behind: Real-worde federated class-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2054-2077 2024.
[28] J. Dong, L.. Wang, 7. Fang, G. Sun, S. Xu, X. Wang, and Q Zhu. Federated class-incremental learning. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[29] X. Dong, S. Q. Zhang, A. Li, and H. Kung. Spherefed: Hyperspherical federated learning. In European Conference on Computer Vision, 2022.
[30] R. Dorfman, S. Vargaftik, Y. Ben-Itzhak, and K. Y. Levy. Docofl: downlink compression for cross-device federated learning. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[31] S. Dou, E. Zhou, Y. Liu, S. Gao, W. Shen, L. Xiong, Y. Zhou, X. Wang, Z. Xi, X. Fan, S. Pu, J. Zhu, R. Zheng, T. Gui, Q Zhang, and X. Huang. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug. 2024.
12

---

[32] Y. H. Ezzeldin, S. Yan, C. He, E. Ferrara, and A. S. Avestimehr. Fairfed: Enabling group fairness in federated learning. In Proceedings of the AAAI conference on artificial intelligence. pages 7494-7502, 2023.
[33] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: (et::80033300 1803.03635, 2018.
[34] J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Pruceedings of the 37th International Conference on Machine Learning, pages 3259- 12599,326,,44 2020.
[35] Geirhos, J.-H. Jacobsen, C. Michaelis, Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):665 673, 2020.
[36] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robust- ness. In International Conference on Learning Representations, 2018.
[37] J. Goetz and A. Tewari. Federated learning via synthetic data. arXiv preprint arXiv:2008.04489, 2020.
arXiv preprint [38] N. Guha, A. Talwalkar, and V. Smith. One-shot federated learning. arXiv:1902.11175, 2019.
[39] I. Gulrajani and D. Lopez-Paz. search of lost domain generalization. In International Conference on Learning Representations, 2020.
[40] C. Guo, M. Rana, M. Cisse, and van der Maaten. Countering adversarial images using input transformations. In ICLR, 2020.
[41] K. Gupta, M. Fournarakis, M. Reisser, C. Louizos, and M. Nagel. Quantization robust federated learning for efficient inference on heterogeneous devices. Transactions on Machine Learning Research, 2023.
[42] F. Hamman and S. Dutta. Demystifying local & global fairness trade-offs federated learning using partial information decomposition. In The Twelfth International Conference Learning Representations, 2024.
[43] W. Hao, M. El- J. Lee, J. Zhang. K. J. Liang, C. Chen, and L. C. Duke. Towards fair federated learning with zero-shot data augmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3310-3319, 2021.
[44] M. Hardt, K. Ligett, and F. Mesherry. A simple and practical algorithm for differentially private data release. In F. Pereira, C. J. C. Burges, I.. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
[45] M. Hardt and G. N. Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pages 61-70, 2010.
[46] C. He, M. Annavaram, and S. Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. In Advances in Neural Information Processing Systems 34, 2020.
[47] C. He, S. Li, J. So. M. Zhang, H. Wang. X. Wang. P. Vepakomma, A. Singh, H. Qiu, L. Shen, P. Zhao, Y. Kang, Y. Liu, R. Raskar, Q. Yang, M. Annavaram, and S. Avestimehr. Fedml: re- search library and benchmark for federated machine learning. arXiv preprint arXiv:2007. 2020.
[48] C. He, L. M. Soltanolkotabi, and S. Avestimehr. Pipetransformer: Automated elastic pipelin- ing for distributed training of large-scale models. Proceedings of the 38th International Conference on Machine Learning, pages 4150-4159. PMLR, 18-24 Jul 2021.
13

---

[49] C. He, A. D. Shah, Z. Tang, D. F. N. Sivashunmugam, K. Bhogaraju, M. Shimpi, L. Shen, X. Chu, M. Soltanolkotabi, and Avestimehr. Fedev: federated learning framework for diverse computer vision tasks. arXiv preprint arXiv:2 1066, 2021.
[50] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[51] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[52] C. E. Heinbaugh, E. Luz-Ricca, and H. Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.
[53] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[54] R. D. Hjelm, A. Fedorov, S. Marchiln,,, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
[55] C.-H. Ho and N. Nvasconcelos. Contrastive learning with adversarial examples. Advances in Neural Information Processing Systems, 10117781-170000 2020.
[56] T. Hsu, H. Qi, and M. Brown. Measuring the effects of non-identical data distribution for federated visual classification. ArXiv, htts/1999.06335, 2019.
[57] T.-M. H. Hsu, H. Qi, and M. Brown, Federated visual classification with real-world data distribution. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision ECCV 2020, pages 76-92, Cham, 2020. Springer International Publishing.
[58] S. Hu, Q. Li, and B. He. Communication-eficce learning. In Proceedings of the 52nd International Conference on Parallel Processing, ICPP '23, page 254-263, New York, NY, USA, 2023. Association for Computing Machinery.
generalized neuron matching for federated
[59] C. Huang, Q. Liu, B. Y. Lin, C. Du, Pang, and M. Lin. Lorahub: Efficient cross-task generalization via dynamic loRA composition, 2024.
[60] W. Huang, M. Ye, Shi, H. Li, and B. Du. Rethinking federated learning with domain shift: prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[61] P. Izmailov, Kirichenko, N. Gruver, and A. G. Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516-38533 2022.
[62] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim. Communication-effiiie on- device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11477) 2018.
[63] Jhunjhunwala. S. Wang, and G. Joshi. Towards theoretical and practical understanding of one-shot federated learning with fisher information. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities. 2023.
[64] Y. Jiang, S. Wang. V. Valls, J. Ko, W.-H. Lee, K. K. Leung, and L. Tassiulas. Model pruning enables efficient federated learning edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[65] X. Jin, X. Ren, D. Preotiuc-Pietroe and P. Cheng. Dataless knowledge fusion by merging weights of language models. The Fleventh International Conference on Learning Represen- tations, 2023.
14

---

[66] N. Johnson, J. P. Near, and D. Song. Towards practical differential privacy for sqi queries. Proceedings of the VLDB Endowment, 11(5):526-533) 2018.
[67] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L.. D'Oliveira, H. Eichner, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Har- chaoui, C. He, L.. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konečný, A. Korolova, F. Koushanfar, S. Koyejo, T. Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, M. Raykova, H. Qi, D. Ramage,
Raskar, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, and S. Zhao. Advances and open problems in federated learning, 2021.
[68] S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh. Scaffold: Stochastic controlled averaging for federated learning. arXiv preprint arXi::/910.063//1 2019.
[69] H. Kim, Y. Kwak, M. Jung, Shin, Y. Kim, and Kim. Protofl: Unsupervised federated learning via prototypical distillation. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
[70] P. Kirichenko, P. Izmailov, and A. G. Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations. 2023.
[71] J. Kirkpatrick, R. Pascanu, N. Rabinowitz,
J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinskk. D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 2017.
[72] J. Konečný, H. Brendan McMahan, F. X. Yu, P. Richtárik, A. Theertha Suresh, and D. Bacon. Federated Learning: Strategies for Improving Communication Efficiency. arXiv e-prints, 2016.
[73] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, 2009.
[74] A. Krizhevsky, V. Nair, and G. Hinton. Cifar- 10 (canadian institute for advanced research). URL http:://wwww...ooooooooe.edadeee
2010.
[75] A. Kumar, A.
Raghunathan, R. M. Jones, T. Ma, and P. Liang. Fine-tuning can distort pretrained features and underperform dittof-distribution. In International Conference on Learning Representations, 2022.
[76] Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[77] Y. Lecun, Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11), 1998.
[78] A. Li, J. Sun, Wang, L. Duan, Li, Y. Chen, and H. Li. Lotteryfl: Empower edge intelli- gence with personalized and communication-effiiienn federated learning. In 2021 IEEE/ACM Symposium on Edge Computing (SEC), pages 68-79, 2021.
[79] D. Li and J. Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581. 2019.
[80] Q Li, Y. Diao, Q Chen, and B. He. Federated learning on non-iid data silos: An experimental study, 2021.
[81] Q Li, B. He, and D. Song. Practical one-shot federated learning for cross-silo setting. arXiv preprint arXiv:2010.01007, 2020.
[82] Q. Li, B. He, and D. Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 10713-10722, 2021.
15

---

[83] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020.
[84] X. Li, Z. Zhou, J. Zhu, J. Yao, T. Liu, and B. Han. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:231 1.0011,, 2023.
[85] Y. Li, B. Luo, Q. Wang, N. Chen, X. Liu, and B. He. A reflective llm-based agent to guide zero-shot cryptocurrency trading. arXiv preprint arXiv:2407.09554. 2024.
[86] Z. Li, X. Shang. R. He, T. Lin, and C. Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5319 -329,, October 2023.
[87] P. P. Liang, T. Liu, L. Ziyin, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L.-P. Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint 2rXiv:2001.01523, 2020.
[88] T. Lin, L.. Kong, S. U. Stich, and M. Jaggi. Ensemble distillation for robust model fusion in federated learning. In NeurlPS, 2020.
[89] J. Liu, Z. Shen, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108. 13624, 2021.
[90] L. Liu, X. Jiang, F. Zheng, H. Chen, G.-J. Qi, H. Huang, and L. Shao. A bayesian federated learning framework with online laplace approximation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[91] Y. Long, B. Wang, Z. Yang, B. Kailkhura, A. Zhang, C. Gunter, and B. Li. G-pate: Scalable differentially private data generator via private aggregation of teacher discriminators. Advances in Neural Information Processing Systems, 34, 2021.
[92] B. Luo, 7. Zhang, Q Wang, A. Ke, S. Lu, and B. He. Ai-powered fraud detection in decentralized finance: A project life cycle perspective. arXiv preprint arXiv:2308. 15992, 2023.
[93] M. Luo, Chen, D. Hu, Zhang, Liang, and J. Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-lID data. In A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[94] Z. Luo, Y. Wang, Z. Wang, Z. Sun, and T. Tan. Disentangled federated learning for tackling attributes skew via invariant aggregation and diversity transferring. In K. Chaudhuri, Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14527-14541. PMLR, 17-23 Jul 2022.
[95] M. S. Matena and C. A. Raffel. Merging models with weighte averaging. Advances in Neural Information Processing Systems, 11777703-7716,, 2022.
[96] B. McMahan, F. Moore, D. Ramage, S. Hampson, and B. A. Arcas. Communication-eficieent learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282, 2017.
[97] L. Melis, C. Song, De Cristofaro, and V. Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy (SP), pages 69 691-70 IEEE, 2019.
[98] V. Mugunthan, E. Lin, V. Gokul, C. Lau, L. Kagal, and S. Pieper. Fedltm: Federated learning for sparse and personalized lottery ticket networks. In European Conference on Computer Vision, pages 65. Springer, 2022.
[99] M. Naseer, S. Khan, M. Hayat, F. S. Khan, and F. Porikli. A self-supervised approach for adversarial robustness. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 262-271, 2020.
16

---

[100] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natu- ral images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011.
[101] Neyshabur, S. Bhojanapalli, D. Mcallester and N. Srebro. Exploring generalization deep learning. In Advances in Neural Information Processing Systems, 2017.
[102] Neyshabur, H. Sedghi, and C. Zhang. What being transferred in transfer learning? Advances in neural information processing systems, 2020.
[103] J. Nguyen, K. Malik, M. Sanjabi, and M. Rabbat. Where to begin? exploring the impact of pre-training and initialization in federated learning. arXiv preprint 2tti:22206.133877. 2022.
[104] N.-H. Nguyen, T.-A. Nguyen. T. Nguyen, V. T. Hoang, D. D. Le, and K.-S. Wong. Towards efficient communication federated recommendation system via low-rank training. arXiv preprint 240iv::2001.334455 2024.
[105] A. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: / 08000037744 2018.
[106) M. Pagliardini, M. Jaggi, F. Fleuret, and S. P. Karimireddy. Agree disagree: Diversity through disagreement for better transferabilltyy In The Eleventh International Conference on Learning Representations, 2023.
[107] N. Papernot and P. McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning, arXiv preprint 1803 04765, 2018.
[108] Y. Park, D.-J Han, D.-Y. Kim, Seo, and J. Moon. Few-round learning for federated learning. Advances in Neural Information Processing Systems, 34:28612-2888 2021.
[109] J. Pearl. Causality. Cambridge university press, 2009.
[110] J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[111] X. Qiu, J. Fernandez- Marques,, P. P. Gusmao, Y. Gao, T. Parcollet, and N. D. Lane. ZeroFL: Et- ficient on-device training for federated learning with local sparsity. In International Conference on Learning Representations, 2022.
[112] Z. Qu, X. Li, R. Duan, Y. Liu, B. Tang, and Z. Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, pages 18250-18280. PMLR, 2022.
[113] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. Sveca: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proceedings the 31st International Conference on Neural Information Processing Systems, NIPS 17, 2017.
[114] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021.
[115] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani. Fedpaq: A coamunication-effiiine federated learning method with periodic averaging and quantiza- tion. In International Conference on Artificial Intelligence and Statistics, pages 2021-2031. PMLR, 2020.
[116] S. Sagawa, A. Raghunathan, P. W. Koh, and P. Liang. An investigation of why overparameteri- zation exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346-83561 PMLR, 2020.
[117] K. Sahu, T. Li, M. Sanjabi, M. Zaheer, Talwalkar, and V. Smith. On the convergence of federated optimization in heterogeneous networks. ArXiv, abs/1812.06117, 2018.
17

---

[118] A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124020. 2019.
[119] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)612-634, 2021.
[120] V. Sehwag, S. Mahloujifar, T. Handina, S. Dai, C. Xiang, M. Chiang, and P. Mittal. Improving adversarial robustness using proxy distributions. arXiv preprint arXiv:2/04.09425, 2021.
[121] M. Shin, C. Hwang, J. Kim, J. Park, M. Bennis, and S.-L.. Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning. arXiv preprint arXiv:2006.08 148, 2020.
[122] R. Shwartz-Ziv and N. Tishby. Opening the black box of deep neural networks via information. 2017.
arXiv preprint
[123] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., (5(1):1929-1555 jan 2014.
[124] Y. Sun, L. Shen, S. Chen, Ding, and D. Tao. Dynamic regularized sharpness aware minimization in federated learning: Approaching global consistency and smooth landscape. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[125] Y. Sun, L. Shen, T. Huang, L. Ding, and D. Tao. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy. In The Eleventh International Conference on Learning Representations, 2023.
[126] A. Z. Tan, H. Yu, L. Cui, and Q Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, pages 1-17, 2022.
[127] Z. Tang, X. Chu, R. Y. Ran, S. Lee, S. Shi, Y. Zhang, Y. Wang, A. Q. Liang, S. Avestimehr, and C. He. Fedml parrot: A scalable federated learning system via heterogeneity-aware scheduling sequential and hierarchical training. arXiv preprint arXiv:2303.01778, 2023.
[128] Z. Tang. J. Huang, R. Yan, Y. Wang, Z. Tang. S. Shi, A. C. Zhou, and X. Chu. Bandwidth-aware and overlap- weighted compression for communication efficient International Conference on Parallel Processing. Gotland, Sweden, 12-15 August 2024.
-efffitttion-fficientt federated learning. In 53rd
[129] Z. Tang, X. Kang, Y. Yin, X. Pan, Y. Wang, X. He, Q. Wang. Zeng, Zhao, S. Shi, A. C. Zhou, B. Li, B. He, and X. Chu. Fusionllm: A decentralized Ilm training system on geo-distributed gpus with adaptive compression, 2024.
[130] Z. Tang, S. Shi, and X. Chu. Communication--ffiiccett decentralized learning with sparsification and adaptive peer selection. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pages 1207-1208. IEEE, 2020.
[131] Z. Tang, S. Shi, X. Chu, W. Wang. and B. Li. Communication-effcce efffccin distributed deep learning: A comprehensive survey. arXiv preprint arXiv:2003.0638 2020.
[132] Z. Tang, S. Shi, B. Li, and X. Chu. Gossipft: decentralized federated learning framework with sparsified and adaptive communication. JEEE Transactions on Parallel and Distributed Systems, pages 1-13, 2022.
[133] Z. Tang, Y. Wang, X. He, L. Zhang, X. Pan, Q. Wang, R. Zeng, K. Zhao, S. Shi, B. He, et al. Fusionai: Decentralized training and deploying lims with massive consumer-level gpus. The 32nd International Joint Conference on Artificial Intelligence, Symposium on Large Language Models (LLM@ @ IJCAI 2023), 2023.
[134] Z. Tang, Y. Zhang, S. Shi, X. He, B. Han, and X. Chu. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In K. Chaudhuri, Jegelka, I. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 21111-21132. PMLR, 17-23 Jul 2022.
18

---

[135] Z. Tang, Y. Zhang, S. Shi, X. Tian, T. Liu, B. Han, and X. Chu. Fedimpro: Measuring and improving client update in federated learning, In The Twelfth International Conference on Learning Representations, 2024.
[136] C. Thapa, M. A. P. Chamikara, S. Camtepe, and L. Sun. Splitfed: When federated learning meets split learning. arXiv preprint arXiv:2004. 12088, 2020.
[137] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning, arXiv preprint arXiv:2005. 10243, 2020.
[138] N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368-377, 1999.
[139] F. Utrera, E. Kravitz, N. B. Erichson, R. Khanma, and M. W. Mahoney. Adversarially-trann trained deep nets transfer better: Illustration image classification, In International Conference Learning Representations, 2020.
[140] B. Vivek and R. V. Babu. Single-step adversarial training with dropout scheduling. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 947-956. IEEE, 2020.
[141] H. Wang. M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni. Federated learning with matched averaging. In International Conference Learning Representations, 2020.
[142] J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXivv2007.077481, 2020.
[143] N. Wang, W. Feng, yuchen deng, M. Duan, Liu, and S.-K. Ng. Data-free diversity-based ensemble selection for one-shot federated learning. Transactions on Machine Learning Research, 2023.
[144] Q. Wang, Z. Zhang, Z. Liu, Lu, B. Luo, and He. Ex-graph: A pioneering dataset bridging ethereum and x. In The Twelfth International Conference on Learning Representations, 2024.
[145] Y. Wang, Z. Ni, S. Song, L. Yang, and G. Huang. Revisiting locally supervised learning: an alternative end-to-end training. In International Conference on Learning Representations, 2020.
[146] B. E. Woodworth, K. K. Patel, and N. Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems, 33:6281-6292. 2020.
[147] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1XXi:1700..07747 2017.
[148] P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal. TIES-mergin Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[149] K. Yang, T. Zhou, Y. Zhang, X. Tian, and D. Tao. Clas--iisentagglemeet and applications in adversarial detection and defense. In NeurlPS, 2021.
[150] Z. Yang, Y. Zhang, Y. Zheng, X. Tian, H. Peng, T. Liu, and B. Han. Fedfed: Feature distillation against data heterogeneity in federated learning, Advances in Neural Information Processing Systems, 36, 2024.
[151] H. Ye, C. Xie, T. Cai, R. Li, 7. Li, and L.. Wang. Towards a theoretical framework of out- of-distribution generalization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[152] M. Yi, L. Hou, J. Sun, L. Shang, X. Jiang, Q. Liu, and Z. Ma. Improved ood generalization via adversarial training and pretraing. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11987-11997, PMLR, 18-24 Jul 2021.
19

---

[153] Н. Yin, P. Molchanov, J. M. Alvarez. Z. Li, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 8715-8724, 2020.
[154] T. Yoon, S. Shin, S. J. Hwang, and E. Yang. Fedmix: Approximation of mixup under mean augmented federated learning. In International Conference on Learning Representations. 2021.
[155] F. Yu, A. S. Rawat, A. Menon, and S. Kumar. Federated learning with only positive labels. In International Conference on Machine Learning, 2020.
[156] B. Yuan, Y. He, J. Q. Davis, T. Zhang, T. Dao, B. Chen, P. Liang, C. Re, and C. Zhang Decentralized training of foundation models in heterogeneous environments. In NeurIPS, 2022.
[157] H. Yuan, W. R. Morningstar, I.. Ning, and K. Singhal. What do we mean by generalization in federated learning? International Conference on Learning Representations, 2022.
[158] D. Zhang, K. Abuja. Y. Xu, Y. Wang, and A. Courville. Can subnetwork structure be the key to out-of-distribution generalization? In International Conference on Machine Learning, pages 12356-12367. PMLR, 2021.
[159] J. Zhang, C. Chen, B. Li, L. Lyu, S. Wu, S. Ding, C. Shen, and C. Wu. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35, 2022.
[160] J. Zhang, Z. Li, B. Li, Xu, S. Wu, S. Ding, and C. Wu. Federated learning with label distribution skew via logits calibration. In Proceedings of the 39th International Conference on Machine Learning. PMLR, 2022.
[161] Y. Zhang, M. Gong, T. Liu, G. Niu, X. Tian, B. Han, B. Schölkopf, and K. Zhang. Ad- versarial robustness through the lens of causality, In International Conference on Learning Representations, 2021.
[162] Y. Zhang, Z. Yang, X. Tian, N. Wang, T. Liu, and B. Han. Robust training of federated models with extremely label deficiency. In The Twelfth International Conference Learning Representations, 2024.
[163] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1800 00582, 2018.
[164] Y. Zhou, G. X. Ma, X. Li, and D. Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.079 07999, 2020.
[165] Z. Zhou, R. Tao, J. Zhu, Y. Luo, Z. Wang, and B. Han. Can large language models reason ro- bustly with noisy rationales? /CLR 2024 Workshop on Reliable and Responsible Foundation Models.
20

---

Appendix supplemental material
A Details of Algorithm
Details of our algorithm is shown Algorithm
Algorithm FL with FuseFL
Input: The number of split modules K; Initialized local modules H ... H and classifier Am. Output:
The fused model Fus ()..6). 6).
# Train and fuse
2: for module k = 1.... K do
3: for each client m E M in parallel do do
4: 8 + -CientTrainnkk),
5: end for
6: Fuse H + -H :..., HM) as Eq. 4
7: for each client m E M in parallel do do
8: ClientAdapt(A,, H
9: end for
10: end for
11: # Calibrate classifier
12: Averaging and calibrating final classifier A using CCVR 93
with object E( 2, y) (Eq. 6);
13: Return Fris.
14:
15: ClientTrain(k):
16: Build Fl.m follows Eq.
17: Freeze H H
18: for each local iteration 0, 0..... E do
0m.j11 +- 0m This Vol(L (r 2 v). x, Dmi
20: end for
21: Return 11 parameterized with 0k m
22:
ClientAdapt(k; Hi:)::
Build adaptor Ak+1 following Eq
Stitch HK+11 A Hk+1 Ak+l,
26:
B Broader Impact
This work provides novel OFL approach, aiming at advancing the field of federated learning. There are many potential societal consequences of our work. For instance, our work can extremely reduce the communication cost of FL., reducing energy consumption. Under the low-bandwidth communication environments like Internet with 1~ 10 MB/s 133 156 129], this method provides possibility train super large models like large language models with low communication time. Furthermore, to the best of our knowledge, this is the first work that understands FL from the view of causality. There exist large space for future works to study along this direction.
FuseFL can further inspire more research and applications in the communication caanmuiccaion-constaaaie constrained sce- narios, like extremely low communication bandwidth, and training large language models (LLM) in FL [12 84] [65]
We will try to extend FuseFL into training LLMs or MoE in FL scenarios with its low communication costs.
To deploy transformer- based frameworks like current L.LMs with the core idea of FuseFL one-shot FL., we envision two methods here.
Concat-and-freeze Similar training ResNet in FuseFL, we can block-wisely train and collect the transformer blocks together for each round; during local training, the output features of all
21

---

transformer blocks are concatenated to feed into the subsequent layers. Due to the large resource consumption of pretraining, we do not evaluate this idea here.
weraging-and-freen LoRA. Here, we consider the finetuning scenarios with LoRA (59] LoRA blocks can be seen as additional matrix mapping applied on the local Q attentions and MLP layers. The output is the original feature plus the LoRA output. To use LoRA in FuseFL, we can follow the MoE style [I or the averaging style 59) Specifically, we consider averaging LoRAs s different clients together, then averaging and freezing all LoRAs in each transformer block to freeze the obtained aggregated features in each communication round.
C More related works
We provide the comprehensive introduction of the related works in this section. The data heterogeneity problem in FL is introduced in Section and the communication compression in FL is introduced in Section C.3 Further, by seeing local datasets from other clients the OOD datasets with respect to the isolated locally trained models, we review some methods in OOD generalization (Section C.5 and mutual information (Section C.4) to build connection between them and FL with extremely low communication costs. We demystify our work from current works in Table g which a detailed version of [
Table 8: Demystifying different FL algorithms. T represents communication rounds, S the model size, M the number of clients. Practically, in commmnicattonnccopperse ratio QSpar is 0, the quantization ratio Qquant is 0.125 (32 + 4bits), and the low rank ratio QLoWR is 0.1, otherwise the convergence is difficult to achieve and significantly harming model performance. The "Centralized" means training the model with all datasets aggregated with SGD, Due the data heterogeneity, the performance usually is: Centralized 2 FedAvg 2 Ensemble.
FL., the minimal sparsification
Communication cost
Performance
Support model heterogeneity
Not require extra data
Method type
Core technique
Storage cost
upper bound
No compression Sparsification
T x S
Centralized
T S x Qspar
S
FedAvg
FodAvg
Quantization
T x S x QQaat
S
FedAvg
Low-rank
T TxSx RSSx x QLoeR
S
FedAvg
Average-based
S
Ensemble
One-shot FL
Ensemble-base
x M
Centralized
Model distillation
S
Ensemble
FuseFL.
5
S
Centralized
C.1 Data Heterogeneity in FL
The Non-IID data distribution is the notorious problem in FL, which severely harms the convergence rate and model performance of FL. The sharp Non-IID data makes local clients learn much different weights [68] [46), resulting in heterogeneous feature 160 134] and classifiers (93). Current methods that address Non-IIID data problems include following typical directions.
Some works design new FL optimizers to stabilize and accelerate convergence 114 125]. Per-
sonalized FL aims to optimize different client models by learning knowledge from other clients and adapting to their own datasets [126] 88). Distinguished from these works, how to alleviate data heterogeneity within extremely low communication costs is a new challenging problem, clients have little possibility of communicating information with other clients.
Model regularization. This direction proposes to add penalty of distances between local trained models and the server model. FedProx [177] directly uses the L2 distance between local models to the server model to constrain the local models not moving too far. SCAFFOLD 688 utilizes the historical local updates to correct update directions local clients during local training. FedDyn [I] dynamically updates the objective functions to ensure the local optima between devices asymptot- ically consistent. FedIR [57] claims that applying important weight to the client's local objectives helps obtain unbiased estimator of the global loss objective function.
Feature calibration. Some works focuses on align feature representations of different clients in similar spaces 82 29 134 60). MOON (82 adds the contrastive loss to between local and global
22

---

models to learn a similar representation between clients, in which the global model acts as an intermediate agent to communicate between clients. It is found the local features of the same data largely shift between client models during local training [134]. To address this problem, virtual homogeneous learning [44 proposes to use a homogeneous dataset which can contain completely no information of original datasets, to calibrate the feature representations between clients. This technique improves the generalization performance and convergence speed of federated learning. SphereFed [29] adds constraints on learned representations of input data to be in unit hypersphere shared by clients. Besides, SphereFed discovers that the non-overlapped feature distributions for the same class lead to weaker consistency of the local learning targets from another perspective. The prototype [60] methods propose to utilize a pre-defined vector of each class in the representation space, then align client feature representations with such prototype. which is also called virtual feature transfer learning [34). Fedlmpro [355] estimates and share the feature distribution to alleviate the gradient diversity and enhance high-level model training.
Classifier calibration. Due to the shifted features between clients, the classifier is usually trained with bias. And the final obtained classifier with FedAvg is normally prone to some specical classes. CCVR [93] is the first work to transmit the statistics of logits and label information of data samples to calibrate the classifier. In SphereFed [29], the classifier is fiexed with weights spanning the unit hypersphere, and calibrated by mean squared loss. Some works also calibrate the classifier during or after training (24 86 69].
Optimization schemes. From the optimization perspective, some methods regard local updates at clients as pseudo gradients [114] and design new FL. optimizers to stabilize and accelerate convergence.
FedNova [42 normalizes the local updates to reduce the inconsistency between the local and global optimization objective functions. FedAvgM [56] exploits the history updates of server model to avoid the overfits on the selected clients in each round. FedOpt [T4) generalizes the centralized optimizers into FL scenario, and proposes FedAdaGrad, Fed FedAdam. FedSpeed [ utilizes a correction term on local updates to reduce the biases during training. FedSpeed also merges the stochastic gradient with perturbation computed from an extra gradient ascent step in the neighborhood, to reduce the gradient heterogeneity.
Data& Feature sharing. The phenomenon of client drift primarily originates from data heterogeneity, To address this issue, researchers have discovered that sharing a subset of private data can markedly enhance convergence speed and generalization capability, as highlighted in [663] However, this approach entails a compromise on the privacy of client data.
Consequently, to simultancously mitigate data heterogeneity and uphold data privacy, a range of studies (45 44 17
85 15) have proposed the addition of noise to data. This method allows for the sharing of data while providing certain level of privacy protection. Alternatively, other research
666
43 or data statistics [12]] [14), as opposed to raw data, in order to alleviate data heterogeneity. FedDF (88) employs external
efforts concentrate on disseminating synthetic data portions 62 91 37
data and engages in knowledge distillation based on these data to facilitate the transfer of model knowledge between the server and clients. The fundamental concept of FedDF involves fine-tuning the aggregated model through knowledge distillation using newly shared data.
Additionally, to address feature shift, certain methodologies advocate for the sharing of features to enhance the convergence rate. Cronus (66] suggests the sharing of logits as means counteract poisoning attacks. CCVR 93 transmits the statistical data of logits to calibrate the final layer of federated models. CCVR [93 also shares parameters representative of local feature distribution. Importantly, this approach does not necessitate sharing the count of different labels with the server, thereby preserving the privacy of clients' label distribution. Furthermore, our method serves as a framework to leverage shared features in reducing gradient dissimilarity. The feature estimator employed need not be confined to the Gaussian distribution of local features; alternative estimators or even features from additional datasets, rather than private ones, may be utilized. In this direction, VHL 134 only requires to share the pure noise dataset, which can have completely no related information of the local private datasets, largely reducing the requirements of the shared datasets and can be exploited in practical scenarios.
Personalized FL. Personalized Federated Learning (PFL) aims to enable clients to optimize distinct personal models that can absorb knowledge from other clients and tailor it to their individual datasets, as detailed in [16). The process of knowledge transfer in personalization primarily
23

---

involves the introduction of personalized parameters [36 78,, or the employment of knowledge
distillation 46 88 79
utilizing shared local features or supplementary datasets.
However, due to the tendency of personalized federated models to prioritize optimizing local objective functions, they often do not achieve generic performance (as evaluated on global test dataset) that is on par with conventional Federated Learning (FL) methodologies [20). Given our primary objective of learning an enhanced generic model, we have chosen not include comparisons and improve performance of personalized FL algorithms in our work. While achieving extremely low communication costs in PFL can be an interesting future work.
C.2
One-shot FL
Current FL methods with communication costs of only one model size are referred one-shot FL, different one-shot learning, the "one-shot" here means one communicating round [159] 38 8T
164 25]. Thus, the communication cost is limited as the model size M for each client, less than
FedAvg-style algorithms for R times. While there are works that introduce additional communication costs to improve the performance of one-shot FL. Existing one-shot FL works can be categorized into avcrage-based [63, ensemble-based 143 or model distillation based 26).
Average-based methods include the basic baseline, one-shot FedAvg, which shows severely bad performance, and some other advanced average methods 195] 65 148.. FedFisher [3] proposes
to utilize fisher information to avoid harming knowledge of local models when averaging on the server. Note that the similar methodology of using fisher information is also utilized to enhance the FedAvg 2 bayesian FL [90 6), which belongs to multi-round FL. All of them aims to avoid forgetting local learned knowledge like the classical exploitation of fisher information in continual learning (71). Matching permutations between the weights of different models (5] [4T] is another advanced method for model averaging. Linear mode connectivity [34] 102 [S] helps to explain to the part of the success of model averaging. However, due the highly non-linear structure of deep neural networks, it is extremely difficult to find method to directly average local clients through one round to obtain a perfect server model. Therefore, one-shot average method usually fails to achieve a good performance.
Ensemble- based methods are based on ensemble learning (38. Intuitively, the server aggregates multiple local client models. Then, the direct way of deploying these models is to average outputed logits of them and make predictions [143. Some methods propose find a better model selection method to select local models that might be more familiar with the given inputs [143]. Some methods
utilize prototype data (26) or generated datasets [52) conduct better model selection in ensemble
learning.
Model distillation methods built upon the ensemble models 38 81. The core motivation is that
the ensemble models occupy too much storage and require much extra computation costs. Thus [8]] involved public dataset for training. As replacement of using global data, [164] transmits the distilled local datasets to server for model distillation.
(25] clusters clients and communicates the mean data of each cluster. These methods require extra datasets, which may be impractical in some data-sensitive scenarios like medical and education data. And the large storage costs still exist when many clients upload their models for ensemble learning.
Note that there is also some work (108) that proposes to find a good initial model of FL in few communication rounds. Thus the convergence rate of FedAvg can be accelerated by this good initialization 103 49 The objective of this work is different from one-shot FL or ours.
C.3
Communication compressed FL
Different from one-shot FL which reduces the communication frequency, communication compression methods aim to reduce the communication size in each round. Typical communication-compress.. methods [31 include sparsification
quantization 115 41 and low-rank decom-
position 104 (22.
have introduced a significant level of sparsity in the local model training stage, effectively reducing the number of parameters that need to be transmitted. The works represented in [ have put forth the Lottery Ticket Hypothesis, suggesting the existence of trainable sub-networks within over paraameeemtt networks that can be independently trained
Sparsification. Studies in 330 12
24

---

without accuracy loss. Inspired by this concept, LotteryFL 78 and FedLTN [9] aim to identify and exchange these personalized lottery ticket networks between the server and clients. Moreover, Hetero-FIAASH [9] employs adaptive sparsity, with objectives extending beyond identifying the optimal sub-network, to fully leveraging clients' resources. contrast to focusing on personalized Federated Learning (FI.), this paper primarily considers generic FL., where all clients share the same model structure.
Quantization. Orthogonally to Sparsification, quantization emerges as an additional pivotal strategy to alleviate the communication bottleneck in Federated Learning. This method represents model updates with fewer bits than the conventional 32 or 64 bits. thus reducing numerical precision. FedPAQ [T15] adopts periodic averaging of low-bit represettttn of local model updates minimize both the frequency of communication and the overhead per round. Advancements in this realm have been furthered by [41], which introduces a variant of Quantization-ANe multiple bit-widths, eliminating the necessity for retraining in the FL. context.
Training (QAT) robust to
Low-rank decomposition. Concurrently, research in [104. 72 has utilized low-rank decompo- sition of matrices to cultivate sparse models. Specifically, the weights of local trained models are decomposed into smaller matrices. Then these smaller matrices are communicated to the server for recovering the original weights.
While these methods also reduce the communication costs, they are far from reducing costs at an extremely low level of the one-shot FL and ours. Because these methods still require many even more communication rounds than FedAvg achieve training convergence.
C.4 Mutual Information
The mutual information (MI) has garerred increasing attention in recent years with its explanation of how deep neural networks learn intermediate representations of the raw data. The Information Bottleneck (IB) principle [122 118 138 provides insights into the training dynamics of deep neural networks. Specifically, the neural network reduces the mutual information between raw data and representations layer by layer, while maximizing the mutual information between labels and repre- sentations. 2 introduces the nuisance variable into the mutual information, and proposes that mutual information between representation and the nuisance should be as less as possible. And it is proved that the empirical risk minimization (ERM) with stochastic gradient descent (SGD) has implicitly achieved the IB principle 2 Some methods in unsupervised learning exploit maxinization of mutual information [105] 137 54] to enhance the feature representation. The contrastive learning [2] 50 maximizes the mutual information across varying views of the same input.
C.5 Out-of-Distribution Generalization
Out-of-Distribution (OOD) generalization refers to the model performance the unseen data distribution, which is called OOD data 889 12 B]. The spurious features widely exist in the real- world datasets, like the textures, shapes, colors of objects [53] 36 35 144 92]. Classification with original labels with these properties often leads overfitting on these variables instead of learning the real mapping relationship X Y. It found that the out-of-distribution generalization performance of a model learned by ERM 22 151 is connected with the spurious features. It is found that the over- over-parametern prer--aarrrriationnn increasing model size well beyond the point of zero training error, may hurt the test error some data samples, while it improving the average test error on all data samples [166.. These aggravated test errors are called worst-group error [8)]
Invariant risk minimization (IRM) 8 is proposed to address inheriting spurious correlations in trained models. is shown that exploiting invariant causal relationships between datasets gathered from multiple environments rather than relying on varying spurious relationships appearing in isolated local datasets, help improve the robustness of the learned predictors. [158] proposes that there exists sub-network hidden in the full neural network that is unbiased functional (not focusing on spurious correlation), thus achieving better OOD performance.
However, some studies show that there is no clear winner between ERM and IRM when covariate shift happens [4) and ERM is still the state-of-the-art on many problems of OOD) generalization (39
The colored MNIST (CMNIST) also called anti-causal CMNIST proposed by [ is often used to study the OOD performance of different training methods. In our work, we directly add different
25

---

geometry shapes with random colors on local clients as the spurious features (also called adversarial explore how isolated local training and our algorithm are influenced by
samples [149 120 55))
them.
It is proven that the self-supervised unsupervised training with contrastive learning 55 99 help reduce the overfitting on spurious features. And dropout [123] also helps learning spurious relationships between label and the spurious features []40)
D Details of Experiment Configuration
D.1 Hyper Parameters
Table 9: Learning rate of all experiments.
Dataset
MNIST
FMNIST
CIFARIO
SVHN
CIFART00
Tiny- Imagennteent
a 0.1
0.001
0.001
0,01
0.001
0.01
0.01
a 0.3
0.001
0.01
0.01
0.01
0.01
0.01
a = 0.5
0.001
0.01
0.01
0.01
0.01
0.01
The learning rate configuration has been listed in Table We report the best results and their learning rates (searched in (0.0001, 0.001, 0.01, 0.1) }). During local updating, the optimizer is SGD with 0.9 momentum following most FL baselines
D.2 Hardware and software
The FedAvg baseline is conducted based on the standard commonly used library FedML [47 127 While other OFL baselines are implemented following [159]. All experiments are conducted based on NVIDIA 2080 Ti or NVIDIA V100 GPU for several hours. Users just need one single GPU to run these experiments.
D.3 Mutual Information Estimation
Due to the extremely high dimension of intermediate features, MI estimation is very difficult [1] Simple neural networks may fail to accurately estimate MI. Thus, we follow [145,, use reconstruction error from h to æ to estimate the (h,,)), and the classification error to estimate ((, y). The details are provided as follows.
Estimating / (h, x). Assume that Rx)(h) denotes the expected error for reconstructing x from h. It H(x)) H(x h) Hx)) R(x(h) where
has been widely known that R(x h)) follows [h,, x))
H (xx denotes the marginal entropy of æ, constant [34] We estimate Ihh, x)) by training decoder
parameterized by w reconstruct the original image x, namely [hh, x) max (x) Ru (z ).
For estimating [hh, æ), the decoders are multiple up-sampling convolutional layers following [45]
Estimating I(h, y). Since T(h, y) -H(y) Hly)h) Hly) E(h, v) p(y[h)), we can directly train an auxiliary classifier (u(())) with parameters ub to approximate p(y(h), such that we have I (h, y) max (H) EA[E, -P(y(4)log do (y]h). To summarize, given the split features H at layer k. we freeze previous layers that with index i S k, and train a new inserted linear layer classifier to estimate T(h,y). For estimating I(h, æ)), the decoder follows [145].
D.4 Linear Separability
Following 7 [OTT, for each layer & to be examined, stitch and train linear classifier M LPP following it, while freezing previous layers. Thus, the linear separability of feature H is shown by the classification error of Mt.tt The MLP is trained by 10 epochs.
D.5 Backdoored Datasets
Figure 5 shows the original images and the backdoored images CIFAR-10. The shapes are added on images according to label indexes but with random colors. By training on backdoored images, the
26

---

local model is easily to fit on the shapes instead of original images. Each shape occupies 10 x 10 image size. Table provides the test accuracy of different methods training with backdoored CIFAR- 10, showing that the performance is severely harmed by the backdoored datasets. While FuseFL, is severely affected by the backdoor.
Figure 5: Each row is class of original (upper) and backdoored (lower) images of CIFAR-10. The shapes are added on images according to label indexes.
E
More Results
To validate the effect of FuseFL when training heterogeneous models with more clients, we further conduct experiments with M = 10 and compare results with M as shown in Table IO] Results show that the FuseFL well supports training heterogeneous models, of which the performance is still comparable with Ensemble FI., but requires much less storage cost than Ensemble.
E.1 Heterogencous Model with Different Number of Clients
Table 10 shows how number of clients influences the performance of FuseFL with average or conv xI as adapter. Results show that with increased M 10, FuseFL still provide benefits to training heterogeneous model.
Table 10: Accuracy with FuseFL with convl 1 or averaging to support heterogeneous model design on CIFAR-I0 with different number of clients.
# Clients
M
M 10
Ensemble
79.91
77.25
FuseFL
81.15
78.28
FuseFL (Avg)
80.35
76.52
FuseFL Hetero
82.71
76.47
FuseFL (Avg) Hetero
79.74
75.90
E.2 Comparisons with FedMA
FedMA 141 does not support training ResNet- 18 and aggregating multi-layers blocks. Thus, we compare it with FedAvg and our methods on training VGG-9 on CIFAR-10. For FedAvg, we run
27

---

it for 10 communication rounds. For ensemble FL methods, the training epoch is set as 200 and communication with only one round. For fair comparison, keep the same computation burden, we divide the 200 epochs to each communication round in FuseFL and FedMA. Thus, FedMA, FuseFL and Ensemble FL. have the same communication costs and 10x less than FedAvg. Note that the FuseFL can have different K' to decide its communication rounds, while FedMA has fixed number communication rounds, i.c. the number layers VGG-9. Thus, the total computation and communication costs of different basclines are same except for FedAvg. Results in Table [I shows that FuseFL successfully outperforms other methods. Note that to achieve the comparable performance to hundreds-of of-roondsrrrrssoooooooooo FedAvg, the FedMA actually requires communicating multiple model size. UNder the limited communication constraints, the matching and averaging ways in FedMA show inferior performance than the feature concatenation and merging as used in FuseFL.
Table 11: Comparing accuracy on CIFAR-10 10 with FedMA.
Algorithm
0.1
0.3
0.5
FedAvg
21.19
26. 16
38.63
FedMA
66.28
71.95
73.34
Ensemble
65.72
67.29
69.58
FuseFL K
73.99
75.58
76.66
FuseFL = 3
72.31
75.29
76.10
FuseFL K 71.34
76.87
75.71
E.3 Different Feature Fusion Methods
Table 12: Accuracy with FuseFL with conv 1 x or averaging adapters of different non-lll IID degree on CIFAR-10.
non-lID degree
0.1
0.3
0.5
Ensemble
57.5
77.35
79.91
FuseFL = 2
70.85
81.41
84.34
FuseFL K
73.79
84.58
81.15
FuseFL = 8
70.46
80.7
74.99
FuseFL (Avg) K 2
70.79
81.5
83.56
FuseFL (Avg) =
68.08
71.49
80.35
FuseFL (Avg) K 8 71.58
81.87
83.29
We verify the effect of using convl x 1 or simply averaging features as feature fusion Table 12 Results shows that using convl 1 is generally better than averaging. With the decreased non-lle II degree, the gap between convl x and averaging is smaller, demonstrating that more similar features require less feature adaptation through learning mapping i.e. training a co.vlxl.
E.4
Communication Cost Comparison
We provide the detailed communication costs of different methods in Table 13 The Other OFLs refer advanced OFL method including DENSE [159], data-free KD methods DAFL 18] and ADI [13] Table [3 shows that FuseFL does not increase the communication costs, while largely improving the model performance.
E,5 Higher Heterogeneity and More Baselines
Table 14 provides results of the higher heterogeneity (a 0.05) and more baselines including DENSE 139] and CoBoosting 233 Results show that the CoBoosting can improve the performance than other baseline methods but fail outperform FuseFL.
Both CoBoosting 23 and FEDCVAF-KD [22 focus on exploiting knowledge distillation methods to improve the performance of global models, while our method focuses how to aggregate models
28

---

Table 13: Communication costs. For different number of clients, the number of basic channels in ResNet- 18 of FuseFL is set as 32, 20, 14, with M E {5, 10, 20, 50), respectively.
# Clients
M 5 5
10
20
50
Single Model
42.66MB
42.66MB
42.66MB
42.66MB
FedAvg (10 rounds)
2133.IMB
4266.2MB
8532.4MB
21331.0MM
Other OFLs
213.31MB
426.62MB
853.24M
2133.10M1
Ensemble
213.31MB
426.62MB
853.24MB
2133. 1133.10MB
FuseFL K = 2
268.55 MB
423.2 MB
842.6 MB
2274.0 MB
FuseFL K
276.9 MB
449.2 MB
944.4 MB
2931.5 MB
FuseFL = 8
340.4 MB
647.7 MB
1722.2 MB
7954.0 MB
FuseFL (Avg)
266.6 MB
416.6 MB
816.6 MB
2109.0 MB
together. Thus, the knowledge distillation is orthogonal with method and may be utilized to enhance FuseFL. For example, one can consider running FuseFL first to obtain fused global model. Then, this model can be used conduct knowledge distillation to guide local model training with the FuseFL once again.
Table 14: Results higher data heterogeneiry (a = 0.05).
Datasets
MNIST
FMNIST
SVHN
CIFAR-10
CIFAR-100
Ensemble
58.06
66.19
62.22
53.33
32,25
FedAvg
46.35
20.07
39.41
17.49
6.45
FedDF
80.73
44.73
60.79
37.53
16.07
F-ADI
80.12
42.25
56.58
36.94
13.75
F-DAFL
78.49
41.66
59.38
37.82
15.79
DENSE
81.06
44.77
60.24
38.37
16.17
CoBoosting
93.93
50.62
65.40
47.20
19.24
FuseFL K 2
95.23
83.23
75.08
46.38
29.98
FuseFL K
95.37
83.65
75.53
51.59
32.71
Limitation
Finding invariant features. With analysis in Section 3 will be useful choose the invariant features Riny during local training for client n. Currently, under the communication and privacy constrains, is further difficult identify which features are spurious. We left this as the future work to explore. We do not explore and exploit this technique in this paper, as it is orthogonal to our core innovation technique, augmenting features by layer-wise model fusion. Nevertheless, by manually crafting spurious correlations by adding backdoored data samples into local dataset, we empirically prove FuseFL can effectively avoid the influences of spurious features.
Limited feature adaptation. In this work, as an initial trial, we only explore using averaging and c the feature adaptation. Further methods consider exploring better feature adaptation methods like using non-linear models.
Security issues. In this work, do not explicitly consider the security issue. However, the vulnerability attacks FuseFL will not be higher than previous multi-round FedAvg, which requires many more communication rounds to achieve the same model performance as FuseFL. For example, FedAvg may require more than 100 rounds to achieve 70% test accuracy as FuseFL with 2 4 rounds, which introduces more communicated information and a higher possibility of attacks. And FuseFL has the same communication size with other OFL methods.
However, while the size of the communication does not increase, FuseFL increases the number of communication rounds compared to other OFI. methods. There are some possible mitigations to address the security issues:
29

---

Adversarial attacks. Some malicious clients might upload adversarial modules or backdoored modules that are used to misguide the aggregated model generate incorrect handcrafted predictions, For these attacks, the possible solution is to detect and reject such malicious uploading also through the lens of causality. Specifically, some images with the invariant features can be fed into the uploaded modules to see whether the output feature can be used to correctly classify images;
Model inversion or Membership attack. Some malicious clients or the server may consider to conduct model inversion or membership attack to obtain the raw data of clients, thus threatening the user privacy. this case, the learned module can be protected with differential privacy to enhance its security.
Multiple communication rounds. To enable concatenating local modules together and training based on aggregated features from these modules, multiple communication rounds are required in FuseFL. this sense, FuseFL belongs to the few-shot FL. However, the communication cost of FuseFL is same as other OFL methods, which is the main claim in the introduction (extremely low communication costs).
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The communication cost and storage costs are compared in Table and [3 The scalability on datasets and clients, computation efficiency, and whether supporting personalized model design have been discussed in Experiment Section
Justification: The limitations have been discussed in Section F
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
31

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We build the causal graph without giving the theorem. The proof of Lemma 3.1 comes from
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the experiment setting, specific baselines (Section 6 hyper- parameters (Section D), the algorithm details (Algorithm A hardware and software details (Section D.2). We have provided our open open-sourced code link.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
32

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided our open- noureed code link. The library has provided the experiment and code instructions. And the datasets are public datasets as listed in Section o which can be downloaded online, or using Pytorch.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https
soiooiiiisiiioiccci.i..oooooiiii for more details. //ipps.ccppubii.. wttpcc/pppppgppu.eeeeeeeeo.opptp httpg//uuieeeeeeeuuiiiieeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the experiment setting, specific baselines, optimiz- ers(Section 6 hyper-parameters (Section D), the algorithm details (Algorithm) A hardware and software details (Section D.2)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do report the error bars in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
33

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the experiment setting (Section 6 and hardware and software details (Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work does not incorporate any ethic concerns of NeurIPS. The datasets and models are commonly used in the community, and the method does not incorporate potential concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
34

---

Justification:
We have discussed the broader impact in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The datasets and baselines, used libraries are properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
35

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.ppperssttt paperssii thcode. com con/datasess package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The datasets and baselines, used libraries are well documented and cited.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
36

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "6Ng7PuqR5w", "reviewer": "Reviewer_mHC6", "summary": "This paper proposes a method FuseFL that aims to perform one-shot FL by progressively fusing models from multiple clients. Their motivation is that within each local client, a model might learn spurious features due to the underlying spurious correlations, adversarial attacks, and shortcuts. However, at the global level, the fusion process would eliminate those spurious features and emphasize on the core/invariant features.\nThey propose a strategy of one-shot fusion of models from multiple clients, followed by experimental results. They also include arguments using mutual information over a Markov chain that use the Data-Processing Inequality.\nExperimental results show benefits over other methods in terms of accuracy of the model.\n\nUPDATE: Increased soundness to 3, Rating to 7 Accept", "questions": "See Weaknesses above.\n\nCould you include a clear notation and definition section where you clarify your notations and independence assumptions on Rspu and other terms at global and local levels? \n\nIf I(Hk; Rspu)=0 is necessary/sufficient?\n\nThis is not really a causality paper. I believe the overreliance on causality is not necessary here.\n\nThe fusion strategy is not properly described. Everything points to one equation (5) which is not described clearly.", "limitations": "Discussed above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses a very interesting problem at the intersection of federated learning and spurious correlations. It is an interesting observation that spurious correlations prevalent at the local model would go away at the global stage by appropriate fusion.\n\nThe idea of one-shot fusion is interesting. They have made an attempt to explain what is going on using information theory and information bottleneck principles.\n\nExperiments compare their strategy with other strategies across several datasets.", "weaknesses": "1. I have some concerns on the clarity of the theoretical analysis here, which if resolved, will increase my rating because I otherwise liked the idea of the paper. I believe Nuisance is mathematically different from Spurious Features (you are assuming them to be the same here)? \n\nNuisance [2] is N which has no information about the true label Y. But, typically in spurious correlation papers, Rspu is something that has information about Y even though they are not causally related, e.g., in Waterbirds dataset, many waterbirds are photographed in front of water background. So, water background (B) naturally has a correlation with bird label (Y) and is not nuisance. But, it is not desirable to use B in the model since it performs poorly on minority groups. Here B is Rspu but not Nuisance.\n\nThe authors also write I(Rspu;H)=0 means spurious feature is not being used. This would also hold for nuisance but not spurious feature as in the example above. One wants the model to not mechanically use the background B. But, what it actually uses instead of the background can still have correlations with the model outputs, and hence mutual information can still be there.\n\nI think what you call Rspu is actually Nuisance at a global level. You actually assume that for the global dataset, Rspu satisfies the definition of nuisance, i.e., being independent with Y, but is dependent at a local level? \n\nCould you include a clear notation and definition section where you clarify your notations and independence assumptions on Rspu and other terms at global and local levels? \n\n2. Experiments: In spurious correlation papers, one is often interested in worst-group/minority group accuracy, i.e., waterbirds on land background. Overall accuracy even decreases sometimes. Could you comment on this in the context of your experiments? How are the local accuracies?\n\n3. There is an overreliance on causality literature including citing Pearl. However, I feel the strategy and the novelty are not that reliant on causality. \n\n4. The fusion strategy is not properly described. Everything points to one equation (5) which is not described clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for taking the time to review our work. We appreciate that you find the solved problem, our observation, and our core idea are very interesting. Please see our detailed feedback for your concerns below.\n\n\n**Q1**: Clarity of theoretical analysis.\n> Rspu is something that has information about Y even though they are not causally related, , e.g., in Waterbirds dataset, many waterbirds are photographed in front of water background. So, water background (B) naturally has a correlation with bird label (Y) and is not nuisance. But, it is not desirable to use B in the model since it performs poorly on minority groups. Here B is Rspu but not Nuisance.\n> The authors also write I(Rspu;H)=0 means spurious feature is not being used. This would also hold for nuisance but not spurious feature as in the example above. One wants the model to not mechanically use the background B. But, what it actually uses instead of the background can still have correlations with the model outputs, and hence mutual information can still be there.\n> I think what you call Rspu is actually Nuisance at a global level. You actually assume that for the global dataset, Rspu satisfies the definition of nuisance, i.e., being independent with Y, but is dependent at a local level?\n\n***Ans for Q1):*** Thanks for your important comments on the theoretical analysis. **Yes, here we call Rspu is actually Nuisance at a global level (global dataset across all clients)**. Thus, on one local client $m$, the spurious feature $R^{spu}_m$ is independent with the $Y^{global}$ in the global dataset, but dependent on the $Y^{m}$ in the local dataset $\\mathcal{D}_m$. I(Hk; Rspu)=0 is a sufficient condition for not overfiting to spurious correlations. Because when I(Hk; Rspu)=0 is not satisfied and Hk takes some spurious features, one can adjust the final classifier to avoid spurious fitting [1,2,3]. Besides, our analysis in Section 3.2 and the Empirical study in Section 3.3 aligns with your understanding.\n\nWe will revise the paper and include the section of clear definitions. **Specifically, we will include the definitions of the local $X_m$, $Y_m$, global $X^{global}$, $Y^{global}$, and the local spurious features $R^{spu}_m$. Furthermore, we will rewrite Lemma 3.1 to make it clear in the federated learning scenario**.\n\n\n\n**Q2**: Experiment issues.\n> In spurious correlation papers, one is often interested in worst-group/minority group accuracy, i.e., waterbirds on land background. Overall accuracy even decreases sometimes. Could you comment on this in the context of your experiments? How are the local accuracies?\n\n***Ans for Q2):*** Thanks for your important questions on the worst-group accuracy. In FL, the local accuracies refer to the accuracy on the local datasets on clients, which is common in personalized FL [4]. Specifically, local test datasets have the similar data distributions to the training datasets (according to the ratio of the labels). One goal of designing experiments with the backdoored Fl CIFAR-10 datasets is to study the majority/minority accuracy. For a backdoored client, images with local spurious shapes and colors become majority group on it (Figure 5 in original paper), while the normal images become minority groups. Now we report the local/global accuracy here. Backdoored (BD) clients fit on the handcrafted spurious features thus having lower global accuracy than normal clients.\n\n| Client | BD-0 | BD-1 | Normal-0 |  Normal-1 | Normal-2 |\n| -------- | -------- | -------- | -------- | -------- |-------- |\n| Local Acc.  |  100.0     | 100.0  | 99.7 | 99.9  | 100.0 |\n| Global Acc.  |  32.6    | 27.1   |  41.2  |  42.3  |  38.4  |\n\n\n**Q3**: Overreliance on causality literature.\n> There is an overreliance on causality literature including citing Pearl. However, I feel the strategy and the novelty are not that reliant on causality.\n\n***Ans for Q3):*** Thanks for your valuable comments. Yes, our strategy does not explicitly adopt the causal discovery to improve one-shot FL. Our motivation is to interpret the mechanism and the root causes of why previous one-shot FL fails through the lens of causality. We appreciate your kind suggestion and will highlight the main contribution in the revision.\n\n\n**Q4**: Unclear fusion strategy.\n> Everything points to one equation (5) which is not described clearly.\n\n***Ans for Q4):*** Thanks for your question. Eq.(5) describes the chain structure of layers in DNNs. For example, a module $H_{12} = H_1 \\circ H_2$, we have the $H_{12}(x) = H_1(H_2(x))$.\n\n\n\n> ***Reference***\n> [1] Class-Balanced Loss Based on Effective Number of Samples. In CVPR 2019.\n> \n> [2] Learning debiased classifier with biased committee. In NeurIPS 2022.\n> \n> [3] Towards last-layer retraining for group robustness with fewer annotations. In NeurIPS 2024.\n> \n> [4] On Bridging Generic and Personalized Federated Learning for Image Classification. In ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mLXYEYZ8Wb", "reviewer": "Reviewer_FVeg", "summary": "The authors provide a causality viewpoint for the data heterogeneity problem in the training of one-shot fed-avg. A block fusing mechanism in the training to provide more information aggregation is designed and the results show a significant improvement for the common tests in ResNet and Cifar.", "questions": "1. what is the definition of \"block\"/\"features\"? I think it will influence the results from the theoretical point of view.\n2. how to deal with the scenes where the disconnection occurs in training, can the system still work (like dynamic topologies in the block communications)\n3. please show the error bar of the experiments since it will not perform significantly better than Ensemble FL for a few of the cases.\n4. I'm also interested in how the model works on the Transformers-based framework, and which part we should aggregate for these models.\nI also wonder if the method can work on the prompt for LLMs.\n5. Please verify the non-IID data-generating processes and whether and how they align with the explanation part.", "limitations": "N/A.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Pros:\n1. the one-shot FL is an interesting and important problem. The trials on the non-IID processing is difficult but significant, which can be further expanded to more industrial scenes.\n2. The idea is well-represented and the performances are great according to the providing verification.", "weaknesses": "Cons:\n1. the cost of the multiple-block communication needs to be considered. In the computation, there might be a huge amount of blocks, we need to keep connections or wait for the block communications in the practical applications.\n2. the explanation part needs to be more rigorous. Please use theorems or propositions for clear statements. I believe the current version can not provide a convincing theoretical explanation from the casual viewpoint.\n3. More large-scale dataset and models need to be considered. The introduction brings a large scope for LLM FL training but not experiments are designed for it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the time in reviewing. We appreciate that you find the problem is interesting, important and practical, our idea is clear and performance is great. Please see our detailed feedback for your concerns below. Some answers can be referred to the global response due to the limited space.\n\n\n**W1 & Q2**: The cost of the multiple-block communication and the disconnection.\n\n***Ans for W1 & Q2):*** Thanks for indicating this practical problem. The synchronous waiting problem also exists in the traditional FL methods like FedAvg and others. Given the number of clients $M$, and model size $S$, the traditional FL methods require communication costs $MSR$, in which the $R$ is the communication round, normally ranging from 100 to 1000. However, our method only requires $MS$ communication cost. For the communication rounds in FuseFL, we can communicate more communicating more layers to reduce the number of blocks (like different $K=[2,4,8]$ in Table 2 and 4 in original paper).\n\nConsider the disconnection, FuseFL needs to wait for the clients. This also happens in the traditional FL. One possible solution is to continue the training process of other clients. After reconnecting, the client loads the newest global model and continue training.\n\n**W2 & Q1**: The explanation part.\n\n***Ans for W2 & Q1):*** Thanks for your important comments. We will revise the paper to make the causality explanation more clear. Specifically, we will rewrite **lemma 3.1 to show how the locally learned spurious $R^{spu}_m$ features influence the mutual information between $H^k_{loc}$ and global $Y$**. The \"block\" means a single or some continuous layers in a DNN. For example, a single or two consecutive convolutional layers in a CNN can be seen as a block. Features refer to the outputs of blocks. For simplicity, the $H^k_m$ represents both block $k$ on client $m$ and features output from it as shown in Figure 1 in original paper. Also, please refer to more explanation of Q1 of Reviewer #mHC6 and the global response.\n\n\n**W3 & Q4**: Large-scale models like LLMs.\n\n***Ans for W3 & Q4):*** Thanks for your important comments. To deploy transformer-based frameworks like current LLMs with the core idea of FuseFL in one-shot FL, we envision two methods here. \n\n- **Concat-and-freeze**: similar to training ResNet in FuseFL, we can block-wisely train and collect the transformer blocks together for each round; during local training, the output features of all transformer blocks are concatenated to feed into the subsequent layers. Due to the large resource consumption of pretraining, we do not evaluate this idea here.\n\n- **Averaging-and-freeze LoRA**: here we consider the finetuning scenarios with LoRA [1]. LoRA blocks can be seen as additional matrix mapping applied on the local Q V attentions and MLP layers. The output is the original feature plus the LoRA output. To use LoRA in FuseFL, we can follow the MoE style [2] or the averaging style [1]. Specifically, we consider averaging LoRAs on different clients together, then averaging and freezing all LoRAs in each transformer block to freeze the obtained aggregated features in each communication round. The following Table shows performance of finetuning Llama2-7B with FuseFL with few-shots FedAvg with OpenFedLLM [3] benchmark (20 clients with Alpaca-GPT4 and MedAlpaca). Results show that FuseFL with much fewer communication rounds can outperform FedAvg of 50 communication rounds.\n\n| Task | MMLU Overall | MMLU Humanity | MMLU Social Science | MMLU STEM | MMLU others |  BBH |\n| - | - |- | - | - | - | - |\n| Base Model  |  38.7    |  36.9  | 48.2 | 32.6 |  44.2 |  34.5 |\n| FedAVG (50 rounds)  |   41.3    | 38.9   |  45.9|  34.8 |  46.9 |  38.9 |\n| FuseFL (4 rounds) | **42.3**    | **39.4** |   **47.6** |  **34.9**    | **48.7** |  **39.6** |\n\n| Task| MedQA | MedMCQA |\n| - | - | - |\n| Base Model  |  21.7    | 20.4 |\n| FedAVG (50 rounds)  |  29.5     | 33.3 |\n| FuseFL (4 rounds)  |  **30.1**    |  **34.2** | \n\n\n\n**Q3**: Error bars.\n\n***Ans for W3 & Q4):*** Thanks for your important comments. We have added the error bars of main experiments as follows and revised the paper. We report a part of them here due to the limited space.\n\n\n| Dataset  |  FMNIST |  | |  CIFAR-10 | |  |\n| - | - |-|-|-|-|-|\n| Heterogeneity | α=0.1 | α=0.3 |  α=0.5 | α=0.1 | α=0.3 | α=0.5 |\n| FedAvg   | 41.69 $\\pm$ 3.58| 82.96 $\\pm$ 2.82         | 83.72 $\\pm$ 2.21| 23.93 $\\pm$ 3.06 | 27.72 $\\pm$ 1.21| 43.67 $\\pm$ 1.84|\n| FedDF    | 43.58 $\\pm$ 3.39  | 80.67 $\\pm$ 2.42| 84.67 $\\pm$ 1.95| 40.58 $\\pm$ 3.72| 46.78 $\\pm$ 1.52| 53.56 $\\pm$ 1.44 |\n| Fed-DAFL | 47.14 $\\pm$ 3.56 | 80.59 $\\pm$ 2.42 | 84.02 $\\pm$ 2.17 | 47.34 $\\pm$ 2.82 | 53.89 $\\pm$ 1.39 | 58.59 $\\pm$ 1.15 |\n| Fed-ADI  | 48.49 $\\pm$ 2.84| 81.15 $\\pm$ 2.29 | 84.19 $\\pm$ 1.71 | 48.59 $\\pm$ 3.18| 54.68 $\\pm$ 1.59| 59.34 $\\pm$ 1.23|\n| DENSE    | 50.29 $\\pm$ 3.15| 83.96 $\\pm$ 2.42| 85.94 $\\pm$ 1.55| 50.26 $\\pm$ 2.52| 59.76 $\\pm$ 1.82 | 62.19 $\\pm$ 1.28 |\n| Ensemble | 67.71 $\\pm$ 1.94| 87.25 $\\pm$ 1.02| 89.42 $\\pm$ 0.57 | 57.5 $\\pm$ 1.82 | 77.35 $\\pm$ 1.21 | 79.91 $\\pm$ 0.63 |\n| FuseFLK=2 | 83.15 $\\pm$ 1.35| 89.94 $\\pm$ 0.72| 89.47 $\\pm$ 0.45| 70.85 $\\pm$ 1.88 | 81.41 $\\pm$ 1.09 | 84.34 $\\pm$ 0.69 |\n| FuseFLK=4 | 83.05 $\\pm$ 1.56 | 84.58 $\\pm$ 0.57 | 90.50 $\\pm$ 0.49 | 73.79 $\\pm$ 1.62 | 84.58 $\\pm$ 0.92 | 81.15 $\\pm$ 0.52 |\n| FuseFLK=8| 83.2 $\\pm$ 1.39 | 88.57 $\\pm$ 0.85 | 88.24 $\\pm$ 0.52| 70.46 $\\pm$ 1.29 | 80.70 $\\pm$ 1.25| 74.99 $\\pm$ 0.91|\n\n\n\n\n\n> ***Reference***\n> \n> [1] LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. In COLM 2024.\n> \n> [2] LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin. In ACL 2024.\n> \n> [3] OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning. In KDD 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LNx0ab2jRC", "reviewer": "Reviewer_V6z4", "summary": "The authors identify the isolation problem as being the root cause of low accuracy in OFL. The isolation problem arises as the clients locally overfit to spurious features observed in their own local datasets in the absence of knowledge from other clients. To better learn invariant features, the authors propose progressive model fusion called FuseFL which fuses client local models block by block. Crucially, this can be done without increasing the total communication cost, however at the cost of additional rounds.", "questions": "The reviewer is positive about this work and is willing to increase the score provided the authors evaluate against Co-Boosting and include a higher heterogeneity level in their overall assessment. While a thorough empirical assessment of the security of the proposed approach is not deemed necessary, the paper must at least discuss the security aspects along with the client side overheads to better inform the readers of the inherent tradeoffs in FuseFL.", "limitations": "No additional limitations than those discussed in the sections above. The authors have well addressed other limitations of their approach involving the memory footprint of the fused model, applicability to model heterogeneity, etc. through their empirical evaluations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors address an important problem in OFL research in a novel way\n- The problem and its solution is clearly presented\n- The experimental evaluation is diverse, in particular, the authors also show the memory cost of fusion and experiments with heterogeneous models", "weaknesses": "- An important baseline [1] is missing from the evaluations which has been shown to improve over the uniform ensemble considered as upper bound in this paper. The CoBoosting algorithm in [1] generates the weights of the ensemble simultaneously with synthetic data to transfer the knowledge to a single global model and is the recent SOTA. Notably, it is a data-free ensemble distillation method in contrast to Table 1.\n- No evaluations are shown with higher heterogeneity levels, such as a = 0.01 or a = 0.05 which is a common setting in OFL papers [1,2,3]. This prevents a comprehensive comparison with prior work.\n- The authors have not discussed important limitations of their work. In particular, the security aspect under multiple rounds of communication. OFL algorithms are recognised not just for their low communication costs but also their lower vulnerability to attacks, thanks to the single communication round. By introducing multiple rounds of communication, FuseFL increases vulnerability to attacks. \n- Similarly, FuseFL appears to increase client side overheads by letting clients run training K times instead of once as in standard OFL. While the computation cost appears to decrease with K, it is nevertheless higher than client side overheads in standard OFL approaches which mostly add server side overheads. Hence, a comprehensive discussion of these limitations is expected to provide a complete picture of tradeoffs.\n\n[1] R. Dai, Y. Zhang, A. Li, T. Liu, X. Yang, and B. Han. Enhancing one-shot federated learning through data and ensemble co-boosting. In The Twelfth International Conference on Learning Representations, 2024. \n\n[2] C. E. Heinbaugh, E. Luz-Ricca, and H. Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.\n\n[3] Jhunjhunwala, D., Wang, S. and Joshi, G., 2024, April. FedFisher: Leveraging Fisher Information for One-Shot Federated Learning. In the International Conference on Artificial Intelligence and Statistics (pp. 1612-1620). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the time in reviewing. We appreciate that you find the problem interesting, important, and practical, our idea is clear and performance is great. Please see our detailed feedback for your concerns below.\n\n**Q1**: Experiment Issues.\n> Important baseline [1] and higher heterogeneity levels.\n\n***Ans for Q1):***  Thanks for your important comments. We have conducted new experiments of CoBoosting [1] and FuseFL with higher heterogeneity levels (a=0.05). The new results are provided following and added in our revision. Results show that FuseFL outperforms CoBoosting and provides higher improvement than the baselines. Note that for CIFAR-10, Ensemble outperforms all baselines but requires extremely large storage and computational costs.\n\n\n| Datasets |  MNIST |  FMNIST | SVHN | CIFAR-10  | CIFAR-100 |\n| --- | --- | -- |  -- |  -- |  -- |\n| FedAvg | 46.35 | 20.07 | 39.41  | 17.49  | 6.45 |\n| FedDF | 80.73 | 44.73  | 60.79   | 37.53 |  16.07|\n| F-ADI  |80.12  | 42.25  | 56.58   | 36.94 | 13.75 |\n| F-DAFL  | 78.49 | 41.66  | 59.38   | 37.82 |  15.79 |\n| DENSE | 81.06 |  44.77  | 60.24   |38.37  |16.17 |\n| CoBoosting | 93.93 | 50.62   |  65.40  | 47.20 | 19.24 |\n| Ensemble | 58.06  | 66.19  | 62.22   | 53.33 | 32.25 |\n| FuseFL K=2 | 95.23 | 83.23  | 75.08   | 46.38 | 29.98  |\n| FuseFL K=4 | **95.37** | **83.65**  | **75.53**   | **51.59** | **32.71**  |\n\n\nAnd we provide following discussion of differences between FuseFL with [1,2,3].\n\n- **Knowledge distillation based FL:** Both CoBoosting [1] and FEDCVAE-KD [2] focus on exploiting knowledge distillation methods to improve the global model performance, while our method focuses on how to aggregate the models together. Thus, the knowledge distillation is orthogonal with our method and may be utilized to enhance FuseFL. For example, one can consider running FuseFL first to obtain a fused global model. Then, this model can be used to conduct knowledge distillation to guide the local model training with the FuseFL once again.\n- **Average-based FL:** FedFisher [3] focuses on how to better average models to obtain a better global model. The fisher information is utilized to identify element-wise averaging weights of parameters in different local models. FedFisher outperforms previous OFL methods [4]. However, FuseFL focuses on a different methodology that concatenates model blocks together instead of averaging. Nevertheless, we consider utilizing the fisher information and average-based methods to enhance FuseFL.\n\nMore differences between FuseFL and Average-based [3,4] and Knowledge distillation [1,2] based OFL can be referred in Table 1 and Appendix C in original paper.\n\n\n\n**Q2**: Security issues.\n> By introducing multiple rounds of communication, FuseFL increases vulnerability to attacks.\n\n***Ans for Q2):*** Thanks for your important comments. Yes, in this work we do not explicitly consider the security issue. However, the vulnerability to attacks of FuseFL will not be higher than previous multi-round FedAvg, which requires many more communication rounds to achieve the same model performance as FuseFL. For example, FedAvg may require more than 100 rounds to achieve 70% test accuracy as FuseFL with $2\\sim 4$ rounds, which introduces more communicated information and a higher possibility of attacks. And FuseFL has the same communication size with other OFL methods.\n\n\nNevertheless, while the communication size is not increased, FuseFL indeed increases the communication rounds than other OFL methods. The possible higher risk of attack is injecting adversarial modules. The possible solution is to **detect and reject such malicious uploading also through the lens of causality**.\n\nFor the model inversion or membership attack, we can consider **adding differential privacy or using noised samples with invariant features** while ensuring the model performance. \n\n\n\n**Q3**: Extra computation overheads.\n> Similarly, FuseFL appears to increase client-side overheads by letting clients run training K times instead of once as in standard OFL. While the computation cost appears to decrease with K, it is nevertheless higher than client-side overheads in standard OFL approaches which mostly add server-side overheads.\n\n***Ans for Q3):*** Thanks for your valuable comments. Sorry for our missed experiment descriptions of FuseFL. We have considered this extra computational problem of repeated local training. To address this problem, we decrease the local training epochs of each client by $K$ times. For example, the FedAvg and other OFL methods require 200 epochs. With $K=4$, for training and merging each block in each round of FuseFL, the local training epochs are decreased to 50. Thus, the number of total training epochs of FuseFL is the same as other OFL methods. The motivation of this design can be referred to the progressive freezing during training DNNS [5]. We have added the descriptions of this design in our revision.\n\n\n\n> ***Reference***\n> \n> [1] R. Dai, Y. Zhang, A. Li, T. Liu, X. Yang, and B. Han. Enhancing one-shot federated learning through data and ensemble co-boosting. In The Twelfth International Conference on Learning Representations, 2024.\n>\n> [2] C. E. Heinbaugh, E. Luz-Ricca, and H. Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.\n> \n> [3] Jhunjhunwala, D., Wang, S. and Joshi, G., 2024, April. FedFisher: Leveraging Fisher Information for One-Shot Federated Learning. In the International Conference on Artificial Intelligence and Statistics (pp. 1612-1620). PMLR.\n> \n> [4] Q. Li, B. He, and D. Song. Practical one-shot federated learning for cross-silo setting. In IJCAI 2021.\n> \n> [5] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. Svcca: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NeurIPS 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SOyJJd0bxK", "reviewer": "Reviewer_twUP", "summary": "This paper proposes FuseFL to address the non-IID data in one-shot federated learning (OFL). Specifically, FuseFL is inspired by the lens of casuality, and fuses each layer of the global model step by step to learn invariant features. Extensive experiments validate that FuseFL achieves SOTA accuracy under various non-IID settings.", "questions": "Please see weaknesses. I think this paper is sound and interesting. I'd like to see more experiments to further verify the effectiveness of the proposed algorithm.", "limitations": "Please see weaknesses.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1: The proposed algorithm is novel and well-motivated.\n\nS2: Extensive experiments validate the effectiveness of FuseFL.", "weaknesses": "W1: Since FuseFL still communicates multiple rounds, I think it belongs to few-shot FL instead of one-shot FL. One-shot FL should have only one communication round. Besides, FedAvg (or other FL algorithms) with the same number of communication rounds should be compared as baselines to ensure a fair comparison. \n\nW2: More experimental setups can be done to further support the effectiveness of the proposed algorithm. For example, in a previous FL benchmark on non-IID data [1], the quantity-based label skew settings are challenging, which are not tested in the current version. Also, in the section of scalability, the authors only test up to 50 clients. I suggest the authors to test on even more clients, e.g. 500 clients, to validate the scalability.\n\n[1] Li, Qinbin, et al. \"Federated learning on non-iid data silos: An experimental study.\" 2022 IEEE 38th international conference on data engineering (ICDE). IEEE, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the time in reviewing. We appreciate that you find the proposed algorithm novel and well-motivated, the experiments are extensive and the FuseFL is effective. Please see our detailed feedback for your concerns below.\n\n**Q1**: Few-shot FL or one-shot FL.\n> Since FuseFL still communicates multiple rounds, I think it belongs to **few-shot FL** instead of one-shot FL. Besides, FedAvg (or other FL algorithms) with the same number of communication rounds should be compared as baselines to ensure a fair comparison.\n\n***Ans for Q1):*** Thanks for your important comments. Yes, FuselFL still communicates with a few rounds. In this sense, FuseFL belongs to the few-shot FL. However, the communication cost of FuseFL is as same as other OFL methods, which is the main claim in the introduction (extremely low communication costs). We will highlight this difference in the introduction of the revised paper.\n\nFurthermore, we conduct new experiments (a=0.1) with 2 and 4 communication rounds of baseline methods with their multi-round versions [2,8] to provide a fair comparison from the round perspective. Note that the FedAvg has communication rounds of 10 for all experiments. Results show that for the same communication rounds, FuseFL still achieves the best performance than other baselines.\n\n| Datasets | CIFAR-10  | SVHN |\n| --- | --- |   -- |\n| FedAvg R=10 | 23.93  | 31.65 |\n| FedDF R=2  | 45.57 | 53.94  |\n| DENSE R=2 |  63.08  | 58.13 |\n| Ensemble |  57.5 |   65.29  |\n| FuseFL K=2 | **70.85**  | **76.88** |\n\n\n\n| Datasets | CIFAR-10  | SVHN |\n| --- | --- |   -- |\n| FedAvg R=10 | 23.93  | 31.65 |\n| FedDF R=4  | 49.16 | 56.29  |\n| DENSE R=4 |  56.26  | 63.95 |\n| Ensemble |  57.5 |   65.29  |\n| FuseFL K=4 | **73.79**  | **78.08** |\n\n\n\n\n**Q2**: Experiment Issues.\n> For example, in a previous FL benchmark on non-IID data [1], **the quantity-based label skew** settings are challenging, which are not tested in the current version. Also, in the section of scalability, the authors only test up to 50 clients. I suggest the authors to test on even more clients, e.g. **500 clients**, to validate the scalability.\n\n\n***Ans for Q2):*** Thanks for your valuable comments. The quantity-based label skew is another kind of heterogeneity different from the Dirichlet sampling based Non-IID simulation. Following your suggestions, we conduct new experiments with the **quantity-based label skew of #C=2** and 10 clients. The test accuracies are shown as follows, illustrating that the #C=2 is a harder setting than Dirichlet with $a=0.1$. While FuseFL can outperform other baseline methods.\n\n\n| Datasets |  MNIST |  FMNIST  | CIFAR-10  |\n| --- | --- | -- |   -- |\n| FedAvg | 23.58  | 23.56   | 21.28 |\n| FedDF | 40.29 | 26.92 | 26.59   |\n| F-DAFL  | 41.59  | 28.45 | 29.83 |\n| F-ADI  | 42.17 | 31.29 | 30.25 |\n| DENSE | 44.2 | 33.53 | 34.83 |\n| Ensemble | 52.3  | 36.23 |38.17 |\n| FuseFL K=2 | 58.74 | 57.72 |  50.79 |\n| FuseFL K=4 | **63.32** | **63.04** | **53.61** |\n\n\nTo validate the scalability of FuseFL, we follow the scale of FL in [2,3] ($M=5\\sim 50$ clients) in original paper. And most OFL methods simulate with less than 50 clients with ResNet and 100 with basic machine learning models [5,6,7]. Simulating 500 clients within the OFL methods is challenging as 500 local models need to be saved to conduct knowledge distillation or ensembling. Loading 500 ResNets into GPU memory is difficult, or offloading and switching between GPU and CPU memory requires much time. To this end, we tried our best to simulate 100 clients to verify the scalability as follows. Results show that FuseFL outperforms all baseline methods except for the Ensemble, which requires significant storage and computation overheads.\n\n| Datasets | CIFAR-10  | SVHN    |\n| --- | --- |   -- |\n| FedAvg | 14.9  | 22.99 |\n| FedDF | 18.15 | 28.53  |\n| F-DAFL  | 20.33  | 33.15  |\n| F-ADI  | 21.56  | 34.29 |\n| DENSE | 22.91  | 34.38 |\n| Ensemble | 39.11 |  44.75  |\n| FuseFL K=4 | **32.92**  | **42.23** |\n\n\n\n> ***Reference***\n> \n> [1] Li, Qinbin, et al. \"Federated learning on non-iid data silos: An experimental study.\" 2022 IEEE 38th international conference on data engineering (ICDE). IEEE, 2022.\n>\n> [2] J. Zhang, C. Chen, B. Li, L. Lyu, S. Wu, S. Ding, C. Shen, and C. Wu. Dense: Data-free one-shot federated In NeurIPS 2022.\n> \n> [3] R. Dai, Y. Zhang, A. Li, T. Liu, X. Yang, and B. Han. Enhancing one-shot federated learning through data and ensemble co-boosting. In The Twelfth International Conference on Learning Representations, 2024.\n> \n> [4] Q. Li, B. He, and D. Song. Practical one-shot federated learning for cross-silo setting. In IJCAI 2021.\n>\n> [5] C. E. Heinbaugh, E. Luz-Ricca, and H. Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.\n> \n> [6] Jhunjhunwala, D., Wang, S. and Joshi, G., 2024, April. FedFisher: Leveraging Fisher Information for One-Shot Federated Learning. In the International Conference on Artificial Intelligence and Statistics (pp. 1612-1620). PMLR.\n> \n> [7] Towards Addressing Label Skews in One-Shot Federated Learning. In ICLR 2023.\n> \n> [8] Ensemble distillation for robust model fusion in federated learning. In NeurIPS 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for taking the time to review our work. We appreciate that you find that we **address a very interesting and important problem in FL** (Reviewer #mHC6, #FVeg, #V6z4), **our observation is novel** (Reviewer #mHC6), **our trials are significant and can be further expanded to more industrial scenes** (Reviwer #FVeg), **our idea and algorithm are interesting and novel** (Reviewer #mHC6, #V6z4, #twUP), **experiments are comprehensive and algorithm is effective** (Reviewer #mHC6, #V6z4, #twUP), the **writing is clear** (Reviwer #FVeg, #V6z4, #twUP).

Here, we provide an overview of responses to main questions for your convenience.

**Q1: More justification on the theory part** (Reviewer #mHC6 and #FVeg)

***Overview of Answers for Q1):*** We provide more clarification of Q1 from Reviwer #mHC6 that the Rspu is actually Nuisance at a global level (global dataset across all clients). We will revise the paper and include a section of clear definitions. **Specifically, we will include the definitions of the local $X_m$, $Y_m$, global $X^{global}$, $Y^{global}$, and the local spurious features $R^{spu}_m$. Furthermore, we will rewrite Lemma 3.1 to make it clear in the federated learning scenario**. We add more explanations about the worst-group/minority group accuracy (Reviewer #mHC6) and the definitions of  "block"/"features" (Reviewer #FVeg) in according responses.


**Q2: More experiments and extensions to more scenarios** (Reviewer #FVeg, #V6z4 and #twUP)

***Overview of Answers for Q2):*** Thanks for all important comments from reviewers. We summarize more experiments we add here.

- **LLM extension** (Reviewer #FVeg): We mainly consider the federated finetuning scenario with LoRA. To use LoRA in FuseFL, we can follow the MoE style [2] or the averaging style [1]. Specifically, we consider averaging LoRAs on different clients together, then averaging and freezing all LoRAs in each transformer block to freeze the obtained aggregated features in each communication round. We follow the FL-LLM setting (20 clients with Alpaca-GPT4 and MedAlpaca) in OpenFedLLM [3] benchmark to conduct limited-rounds FL. The results (Table in response to Reviewer #FVeg) show that FuseFL with much fewer communication rounds can outperform FedAvg of 50 communication rounds.
- **Higher heterogeneity and one more recent advanced baseline** (Reviewer #V6z4): We provide more experiments with a=0.05 to compare the FuseFL with other FL algorithms. A new recent OFL baseline method CoBoosting is added for comparison. Results (Table 2 in response to Reviewer #V6z4) show that FuseFL outperforms CoBoosting and provides higher improvement than the baseline methods. 
- **Multi-round baselines** (Reviewer #twUP): From the perspective of the same communication rounds, we report the results (Table 1&2 in response to Reviewer #twUP) of multi-round versions of baseline methods. Results show that for the same communication rounds, FuseFL still achieves the best performance than other baselines.
- **Another label skew** (Reviewer #twUP): We conduct new experiments with the **quantity-based label skew of #C=2** and 10 clients. The test accuracies are shown in Table 3 in response to Reviewer #twUP), illustrating that the FuseFL can outperform other baseline methods.
- **More clients** (Reviewer #twUP): To validate the scalability of FuseFL, we try our best to increase the number of clients to 100 in OFL methods. Results (Table 4 in response to Reviewer #twUP) show that FuseFL outperforms all baseline methods.


**Q3: Security issues** (Reviewer #V6z4)

***Overview of Answers for Q3):*** In this work, we do not explicitly consider the security issue. However, the vulnerability to attacks of FuseFL will not be higher than the previous multi-round FedAvg, which requires many more communication rounds to achieve the same model performance as FuseFL. And FuseFL does not increase the communication costs.

Nevertheless, compared with other OFL approaches with one single communication round, the number of communication rounds is increased in FuseFL. To this end, we propose several possible solutions to enhance the security of FuseFL:

- **Adversarial attacks**: we can detect and reject such malicious uploading also through the lens of causality.
- **Model inversion or membership attack:** We can consider adding differential privacy or using noised samples with invariant features while ensuring the model performance. 

**Q4: Extra overheads** (Reviewer #V6z4)

***Overview of Answers for Q4):***  We have considered the extra computational problem of the repeated local training with $K$ blocks. To address this problem, we decrease the local training epochs of each client by $K$ times. For example, the FedAvg and other OFL methods require 200 epochs. With $K=4$, for training and merging each block in each round of FuseFL, the local training epochs are decreased to 50. Thus, the number of total training epochs of FuseFL is the same as other OFL methods.

Meta-review: The paper identifies and handles a problem in federated learning where local clients overfit to spurious features. The reviewers found this view, specifically the connection between federated learning and causality, to be interesting and novel. There were some concerns regarding missing experiments but these were mitigated in the rebuttal phase. There is a near concesus that the provided experiments comparing the method to SOTA baselines are thorough, and present a convincing empirical proof of the value of the proposed method. The theoretical analysis was also appreciated by the reviewers, though they did have comments for how to improve its clarity (see e.g. mHC6). Overall, the method seems to be novel and empirical results convincing, this paper should be a good addition to Neurips.
Decision: Accept (spotlight)