FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation
Authors: Xiang Liu
Liangxi Liu
Feiyang Ye
Yunheng Shen
Xia Li
Linshan Jiang
Jialin Li
Venue: NeurIPS 2024 poster (2024)
Keywords: One-shot Federated Learning
Primary area: other
TL;DR: We propose FedLPA to significantly improve the performance via Layer-Wise Posterior Aggregation in one-shot federated learning.

Abstract: Efficiently aggregating trained neural networks from local clients into a global model on a server is a widely researched topic in federated learning. Recently, motivated by diminishing privacy concerns, mitigating potential attacks, and reducing communication overhead, one-shot federated learning (i.e., limiting client-server communication into a single round) has gained popularity among researchers. However, the one-shot aggregation performances are sensitively affected by the non-identical training data distribution, which exhibits high statistical heterogeneity in some real-world scenarios. To address this issue, we propose a novel one-shot aggregation method with layer-wise posterior aggregation, named FedLPA. FedLPA aggregates local models to obtain a more accurate global model without requiring extra auxiliary datasets or exposing any private label information, e.g., label distributions. To effectively capture the statistics maintained in the biased local datasets in the practical non-IID scenario, we efficiently infer the posteriors of each layer in each local model using layer-wise Laplace approximation and aggregate them to train the global parameters. Extensive experimental results demonstrate that FedLPA significantly improves learning performance over state-of-the-art methods across several metrics.

FedLPA: One-shot Federated Learning with Layer-Wise Posterior A Aggregation
Xiang Liu Liangxi Liu Feiyang Ye², Yunheng Shen', Xia Li Linshan Jiang Jialin Li
National University of Singapore, "Nottheastern University,
University of Technology Sydney, 'Tsinghua University, 'ETH Zurich
(liuxiang,1i 1iil1/coooo.p......... nus. edu.ag liu. www.iinngcnorrhhssten.ccom
feiyang-ye ye.esyyy.uuuuamall......
shenyh19@mails. tsinghua. edu. ethlixia@gmail com
linshan@nus.eer edu.
Eqqual Contribution *Correspondence Author
Abstract
Efficiently aggregating trained neural networks from local clients into global model on server is a widely researched topic in federated learning. Recently, mo- tivated by diminishing privacy concerns, mitigating potential attacks, and reducing communication overhead, one-shot federated learning (i.e., limiting client-server communication into single round) has gained popularity among researchers. However, the one-shot aggregation performances are sensitively affected by the non-identical training data distribution, which exhibits high statistical heterogeneity in some real-world scenarios. To address this issue, we propose novel one-shot ag- gregation method with layer-wise posterior aggregation, named FedL.PA. FedL.P aggregates local models to obtain more accurate global model without requiring extra auxiliary datasets exposing any private label information, e.g., label distri- butions. To effectively capture the statistics maintained in the biased local datasets in the practical non-lID scenario, we efficiently infer the posteriors each layer in each local model using layer-wise Laplace approximation and aggregate them to train the global parameters. Extensive experimental results demonstrate that FedLPA significantly improves learning performance over state-of- the-art methods across several metrics.
Introduction
have grown to be major global concern 8 To safeguard data privacy, the conventional federated learning algorithm will use the aggregation methods and follow the data management rules of different institutions, which implies that the distribution of data exhibits variations among clients [8) In the domain of machine learning, federated learning (FL) (9) I0 II] has emerged as prominent paradigm. The fundamental tenet of federated learning revolves around sharing machine learning models derived from decentralized data repositories, as opposed to divulging user raw data. This approach effectively preserves the confidentiality of individual data.
Data privacy issues in Deep Learning 12 5
The standard federated learning framework, FedAvg [2], applies local model training. These local models are then aggregated into a global model through parameter averaging. Existing FL algorithms, however, require many communication rounds to effectively train global model, leading to substantial communication overhead, increased privacy concerns, and higher demand for fault tolerance throughout the rounds. One-shot FL, which reduces client-server communication into single round as explored by prior work [13] [4] is promising yet challenging scheme to address these issues. One-shot FL proves particularly practical scenarios where iterative communication
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

not feasible. Moreover, reduction in communication rounds translates to fewer opportunities for any potential eavesdropping attacks.
While one-shot FL shows promises, existing approaches often grapple with challenges such as inadequate handling of high statistical heterogeneity information [16] 17 non-independent and non-identically distributed (non-lID) data [8] 19]. Moreover, some prior methods rely on an auxiliary one-shot FL
B3 or even on pre-trained large models [20] which may not be practical 21 in some sensitive scenarios. Additionally, some
public dataset to achieve satisfactory performance
19 23 I5I), might expose private label information to both local and global models, e.g., the client label distribution, potentially violating General Data Protection Regu- lation (GDPR) rules. Furthermore, some prior methods [14 [8] [4] require substantial computing resources for dataset distillation, model distillation, or even training a generator capable of generating synthetic data for second-stage -tage training on the server side, making them less practical.
approaches, such as those [22
Besides, the performance of one-shot FL. often falls short when dealing with non-llI) data. Non-III) data biases global updates, reducing the accuracy of the global model and slowing down convergence. In extreme non-lID cases, clients may be required to address distinct classes solely on their side. Sev- eral approaches to federated learning are proposed in multi-round settings tackle this heterogeneity among clients. In the work (25). it allows each client to use personalized model instead of shared global model. With the personalized approach, a multi-round framework benefits from joint training while allowing each client to keep its unique model. However, one-shot aggregation on a local model far from being resolved to address the concern of non-i.i.d data distributions.
In this paper, we introduce a novel one-shot aggregation approach to address these issues, named FedLPA (Federated Learning with Layer-wise Posterior Aggregation). FedLPA infers the posteriors of each layer in each local model using the empirical Fisher information matrix obtained by layer-wise Laplace Approximation. Laplace Approximations are widely used compute the empirical Fisher information matrix for neural networks, conveying the data statistics in non-i.i.d settings. However, computing empirical Fisher information matrices of multiple local clients and aggregating their Fisher information matrices remains an ongoing challenge [17]. To mitigate it, FedLPA aggregates the posteriors of local models using the accurately computed -diao-da empirical Fisher information matrices to measure the parameter space. This matrix captures essential parameter correlations and distinguishes itself from prior methods by being non-diagonal and non-low-rrnk, thereby conveying the statistics of biased local datasets. After that, the global model parameters are aggregated without any need for server-side knowledge distillation 26
Our extensive experiments verify the efficiency and effectiveness of FedLPA, highlighting that FedLPA markedly enhances the test accuracy when compared existing one-shot FL baseline approaches across various datasets. Our main contributions are summarized as follows:
To the best of our knowledge, we are the first to propose an effective one-shot federated learning approach that trains global models using block-diagonal empirical Fisher infor- mation matrices. Our approach is data-free without any need for any auxiliary dataset and label information and significantly improves system performance, including negligible communication cost and moderate computing overhead.
We are the first to train global model parameters via constructing a multi-variate linear objective function and optimizing its quadratic form, which allows us to formulate and solve the problem in a convex form efficiently, which has a linear convergence rate, ensuring good performance.
We conduct extensive experiments to illustrate the effectiveness of FedLPA. Our approach consistently outperforms the baselines, showcasing substantial improvement across various settings and datasets. Even in some extreme scenarios where label skew is severe, e.g., each client has only one class, achieve satisfactory results while other existing one-shot federated learning algorithms struggle.
2 Background and related works
2.1 Federated learning non-iid data
Previous work FedAvg [9 first introduced the concept of Fl. and presented the algorithm, which
2

---

achieved competitive performance on i.idd data, in comparison to several centralized techniques. However, it was observed in previous works [27 28 that the convergence rate and ultimate accuracy
of FedAvg on non-Ill) data distributions were significantly reduced, compared to the results observed
with homogeneous data distributions.
Several methods have been developed to enhance performance in federated learning against non-lID data distributions. The SCAFFOLD method [29] leveraged control variates to reduce objective inconsistency in local updates, It estimated the drift of directions in local optimization and global optimization and incorporated this drift into local training to align the local optimization direction with the global optimization. FedNova [30] addressed objective inconsistency while maintaining rapid error convergence through normalized averaging method. It scaled and normalized the local updates of each client based on the number of local optimization steps. FedProx [i]] enhanced the local training process by introducing a global prior in the form of an 1.2 regularization term within the local objective function. Researchers introduced PFNM 32 33 Bayesian probabilistic framework specifically tailored for multilayer perceptrons. PFNM employed a veta-Bernouulii process (BBP) 34 to aggregate local models, quantifying the degree of alignment between global and local parameters. The framework [17] proposed utilized a multivariate Gaussian product method to construct global posterior by aggregating local posteriors estimated using an online Laplace approximation. FedPA [16] also applied the Gaussian product method but employed stochastic gradient Markov chain Monte Carlo for approximate inference of local posteriors. DAFL (Data- Free Learning) [35] introduced an innovative framework based on generative adversarial networks. ADI (36] utilized image synthesis method that leveraged the image distribution to train deep neural networks without real data. The pFedHN method 37 incorporated HyperNetworks 38 to address federated learning applications.
However, all of these methods encountered challenges in the one-shot federated learning setting, as they required aggregating the model by multiple rounds and might be inaccurate due to the omission of critical information, such as posterior joint probabilities between different parameters.
2.2 One-shot federated learning
One-shot Federated Learning (FL) is an emerging and promising research direction characterized by
its minimal communication cost. In the first study one-shot FL [3), the approach involved the
aggregation of local models, forming an ensemble to construct the final global model. Subsequently,
knowledge distillation using public data was applied in the following step. FedKT [14] brought
forward the concept of consistent voting to fortify the ensemble. Recent research endeavors [9 24
proposed data-free knowledge distillation schemes tailored for one-shot FL. These methods adopted the basic ensemble distillation framework as FedDF (26). Xo.MixFL. (22) introduced the use of exclusive OR operation (XOR) for encoding and decoding samples in data sharing. It is important to note that XorMixFL assumed the possession of labeled samples from global class by all clients and the server, which might not align with practical real-world scenarios. A noteworthy innovation of DENSE [24] was its utilization of a generator to create synthetic datasets on the server side, cireumventing the need for a public dataset in the distillation process. Co-Boosting 39 improves the ensemble when doing the distillation to improve the performance. FedOV [15] delved into addressing comprehensive label skew cases. FEDCVAE (33 confronted this challenge by transmitting all label [5] exposed some client-side private information, leading to additional communication overhead and potential privacy leakage, e.g., FEDCVAE 23 needed all the client label distribution to be transmitted to the server side and FedOV [15 needed the clients to know the labels which were unknown. Instead, MA-Echo [40] adopted unique approach by emphasizing the addition of norms among layer-wide parameters during the aggregation of local models. The project 41 focused on the theoretic analysis of the error in its approximation method. However, their method grappled with limited experiments and lacked detailed explanations of the approach. FedDISC (20). on the other hand, relied on the pre-trained model CLIP from OpenAI, where their reliance might not always align with practicality or suitability for diverse scenarios.
distributions from clients to servers. These schemes [22 14 19
24
23
While some of these techniques are orthogonal to FedLPA and can be integrated with it, it is worth noting that none of the previously mentioned algorithms possess the capability to train global model parameters using empirical Fisher information matrices on extensive experiment settings. Some
3

---

of them [3 14 may require additional information, and may potentially entail the risk of label
distribution leakages.
3 Methodology
3.1
Objective formulation
Generally, federated learning is defined as optimization problem 31
30 29
42 for maximizing
global objective function (0) which is a mixture local objective functions (0, Dx):
K
F(O) Σ
F& (0, Dx)
(1)
WT) ........ vec (W)) is the parameter vector of global model and Wi is the weight and bias of layer for a T--layers neural network; Dj is the local dataset k-th client. (0, Dx) is the expectation of the local objective function, which proportional to the logarithm of likelihood ((x
where |vec W 1)..... vec(W,)
give a common formula of the global posterior which consists of local posteriors p() under variational inference formulation.
Previous works T6 17
K
p(0
II
p(xx 0) II ((DA)
(2)
Del [D]
K
max F 0
-
Σ
E.DD log 0)] max II P(O)as) k=1
As we know, the objective function the expectation of the likelihood, and the sum of the logarithms is equal to the logarithms of the product Eq. 3l Therefore, globally variational inference using Eq. is equivalent to optimization for Eq. 1 Correspondingly. we have:
max (0, max p(0)Da) 0
Following the same training pattern of federated learning, each client infers the local posterior p(0)Dx) by using the local dataset Dj. As a result, the server obtains the global posterior ( by aggregating local posteriors using Eq. [
However, both the global and local posterior usually intractable because modern neural networks are usually non-linear and have a large number of parameters. Therefore, it is necessary to design an efficient and accurate aggregation method for one-shot federated learning.
3.2 Approximating posteriors
Although the posterior is usually intractable, the posterior can be approximated as Gaussian distribution by performing a Taylor expansion on the logarithm of the posterior 43
p( (D log (0" D) (0 T H 0*)
where 0* is the optimal parameter vector, H E.ED H is the average Hessian of the negative log posterior over a dataset D. It is reasonable to approximate global and local posteriors as multi- variates Gaussian distributions with expectations ç 0* and μ4 0 co-variances E -H and
Σg H
M ( = Ni, (i, ()()() 0 M(u,, Σx)
(6)
As result, if given local expectation μ4 and local co-variance Eg. the global posterior is determined by Eq. as below:
f
Σ
E pk:
1-1
Σ

---

Modern algorithms A5 46 allow the local training process to obtain an optimal, regarded as the expectation μ4 the above equations. However, H₃ intractable to compute due large number of parameters in modern neural networks. An efficient method is to approximate H2 using the empirical Fisher information matrix 47.
3.3 Inferring the local layer-wise posteriors with the block-diagonal empirical Fisher information matrices
A empirical Fisher F is defined as below:
= Σ V) sED
(8)
where p(s ()) is the likelihood on data point It is approximate of the Fisher information matrix, the empirical Fisher information matrix is equivalent to the expectation of the Hessian of the negative log posterior if assuming p(s(0) is identical for each 'D
Therefore, the local co-variance Σg can be approximated by the empirical Fisher Fx 48
49
Σx = F A
(9)
The works 50 5
ignore co-relations between different parameters and only consider the self-relations of parameters as computing all co-relations is impossible. Thus, their methods inaccurate. Detailed discussions and the novelty compared to previous works are in Appendix B
In order capture co-relations between different parameters efficiently, previous works 46 43
estimate a block empirical Fisher information matrix instead of assuming parameters are inde- pendent and approximating the co-variance by the diagonal of the empirical Fisher. As pointed out, co-relations inner a layer are much more significant than others [46 52 531, while computing the co-relations between different layers brings slight improvement but much more computation [54] 43). Therefore, assuming parameters are layer-independent is good trade-off. As a result, the approxi- mated layer-wise empirical Fisher is block-diagonal. For layer client k, its empirical Fisher Fx, is one of the diagonal blocks in the whole empirical Fisher for the local model and is factored into two small matrices as below,
Σg R Fú, Ak B
(10) VAI
where & is the Kronecker product; A = [ T ATVAL and BR: =E b bJ
m
are two expectation factor matrices over the data samples; â,, is the activations and be, is the gradient of the pre-activations of layer I on client À is the hyperparameter and is factor minimizing approximation error in Fx. 46 45. Ak, and BN, are symmetric positive definite matrices 45 [6]
use O2, to denote the parameter vector of layer and MA, = vee - (px, ) the vectorized optimal weight matrix of layer on client k. Thus, the resulting local layer-wise posterior approximation is Oz. N(K,,, ).
3.4 Estimating the global expectation
Given the local posteriors, the global expectation could be aggregated by Eq. layer's ' global expectation çi consists of Kronecker products:
With Eq. 10
the I-th
Σ
Σ -1 M4g
E, Σ (As & M4
(11)
K
Σ
veo B Ma, Abt) Σ, Zk. Σ1
where = ΣK 2₁, and zg, vee the global expectation, we have f Σ z. The corresponding global co co-variance is an inverse of the
vec(B B M Ak) ) is immediate notations for simplification. For
5

---

sum of Kronecker products:
K
Σ/
A @ B
(12)
As shown in Eq. obtaining the global expectation μ4 requires calculating the inverse of Σ as Eq. [2] which is unacceptable and the details are in Appendix C Thus, we propose our method to directly train the parameters of the global model on the server side.
3.5 Train the parameters of the global model
We use [A] denotes Previous works [46 49 approximate the expectation of Kronecker products by Kronecker product of expectations E [A & B ss IE [A] @ E [B] with an assumption of Ax, and BN, are independent, which called Expectation Approximation (EA). However, it may lead to a biased global expectation. The details are discussed in Appendix D Instead, we could construct a linear objective after aggregating the approximation of local posteriors via using block boock-diagonal empirical Fisher information matrices. We denotes M as the matrix formula of f = vec(M), and the optimal solution of f is f = vec(MM). We construct (fz) multi-variates linear objective function. When jı fx" is optimal solution, f() = o, where is a vector with all zero. Note that
(Aa). E [B] denotes Σx (Bx). E
B) denotes Ex (Ax & Bx).
(()) = Σー1 Σ vec vec B&MA,)
(13)
vec(E [BMA])
To obtain the optimal solution, minimize the following problem to obtain approximate solution M M:
M min M 2
Σ vec (B&MAs)
(14)
The above equation is a quadratic objective, and can be solved by modern optimization tools efficiently and conveniently. Since the main objective of the above problem is both convex and Lipschitz smooth w.r.t vec(M), we can use the gradient descent method to solve with a linear convergence rate. Here, we automatic differentiation to calculate the gradient w.r.t. M.
Algorithm FedL.PA Global Aggregation
1: Input: clients K, layers L
8: Server executes:
2: Initialize global weight W, of layer 1,.... I,
for
I 1, .... L Lo
10:
A
K
Ak:
3: clients executes:
11:
B; + Σ
B
4: Initialize local model
5: for k 1, .... K do
12:
Ze + Σx B Mr, A
6: [M&, Ak, Bryl 1,... .... L} local
13:
Mi Train the parameter of the global
training
model
7: end for
14:
W M
15: end for
3.6 Overall FedLPA algorithm and discussions
In summary, the proposed algorithm FedLPA follows the same paradigm as the standard one-shot federated learning framework. In FedLPA, the clients locally train their models to get M; and calculate the local co-variance over its training dataset using the layer-wise Laplace approximation to compute Ag, Bg. Subsequently, each client transmits their local Ag. BN, M2 to the server. Following Algorithm[] []] the server aggregates these contributions to obtain the global expectation, as described in Eq. 7 then trains the global model parameters, outlined in Eq. [4 Thus, the transmitted data

---

between the clients and the server is solely As, Bk. M2 without any extra auxiliary dataset and label information.
Note that Fedl.P PPA can be directly adopted in most common scenarios. For the special case that the neural model has enormous single-layer weight parameters, how extend our proposed FedLPA is discussed in Appendix
3.7 t-SNE observation and discussions
To quickly demonstrate the effectiveness of FedLPA, show the t-SNE visualization of our FedLPA global model on the MNIST dataset as an example with biased training data setting among 10 local clients. The experiment details, t-SNE visualizations of the local models and the global models of other algorithms and discussions are in Appendix G.T] As shown in Figure FedLPA generates the global model which can clearly distinguish these classes, meanwhile, the classes are separate.
3.8 Privacy Discussions
FedLPA is intuitively compatible with existing privacy- preserving techniques, such differential privacy (DP)
56 57.. secure multiparty computation (SMC) 588 59],
and homomorphic encryption (HE) (60 61
62 In Ap-
60-
pendix we propose a naive DP-FedLPP with two dif- ferent mechanisms to show the compatibility with differ- ential privacy. Meanwhile, we mention that our proposed FedLPA has the same privacy-preserving level as the con- ventional federated learning algorithms (i.e. FedAvg, Fed- Prox, FedNova and Dense). Compared with FedAvg, we have conducted detailed analysis from privacy attack perspective show that our proposed FedLAP exhibits security level consistent with FedAvg against several types of privacy attacks, where the details are shown in Appendix F3 Note that the main focus of FedLPA is to improve the learning performance on the one-shot FL settings, thus, we leave the integration with other privacy- preserving techniques beyond DP as an open problem.
40
20
-20
-60
-00
-40
-20
20
40
Figure 1: t-SNE visualization for our FedLPA global model.
Experiments
4.1 Experiments settings
Datasets, We conduct experiments on MNIST 63), Fashion-MNIST (64,, CIFAR-10 655, and SVHN [66] datasets. In most of the previous works and the most popular benchmark, the majority of their experiments use these datasets and these models. We choose these datasets and models to do the majority of our experiments following these established methods and benchmarks fairly compare our method with the baselines. We use the data partitioning methods for non-lII settings of the benchmark to simulate different label skews. Specifically, we try two different kinds of partition: 1) #C k: each client only has data from & classes. We first assign & random class IDs for each client. Next, we randomly and equally divide samples of each class their assigned clients; Pk Dir(3): for each class, we sample from Dirichlet distribution Pı and distribute Pk.j portion of class t: samples client j. In this case, smaller denotes worse skews.
Here's a brief overview of these datasets. MNIST Dataset: The MNIST dataset comprises binary images of handwritten digits. It consists of 60,000 28x28 training images and 10,000 testing images. FMNIST Dataset: Similar to MNIST, the FMNIST dataset also contains 60,000 28x28 training images and 10.000 testing images. SVHN Dataset: The SVHN dataset includes 73,257 32x32 color training images and 10,000 testing images. CIFAR-10 Dataset: CIFAR-IO consists of 60,000 32x32 color images distributed across ten classes, with each class containing 6,000 images. The input dimensions for MNIST, FMNIST, SVHN, and CIFAR-10 are 784, 784, 3,072, and 3,072, respectively.
bbttps//githuub..om/Xttt Cwwpr///Xww.o.oooononnD/DD----

---

Table 1: Comparison with various FL algorithms in round.
Danaee
Parition
FeaLMA
FedNova
SCAFFOLD
FodAvg
Fedlbox
DENSE
8=001
21 70:0066
10. 00.13:0000
15 15:97:0111
18.17=0.15
13.37
15234:
18674:041
6=0
03.33:10006
3047::0.39
25
30.93=0.58
51.00:0.:22
52.9142067
9-0.3
68 00:20004
49 4.400.0006
46:00:0002
45.17=0005
04:00:2008
64:2720008
FMNIST
8:0.5
73.33:10.06
57.00:1:288
56.03:1:0288
59,10-0.6
58, 18.10:10.47
12
76.03:0005
63.63-0-33
10:10:0
63.10:1:29
13.20:2:022
10:374:000
10:00::0000
10.37-0:00
1300200118
100010000
#C-2
46.13 13:1..15
21.00:10.10
23.5 22.53:10.22
23.20 0.08
19.97:10.11
38.01:045
PC=3
57.90:1006
27
29.20-0:03
23.93:0.33
53,40:00077
[-00
17:000
11.5740022
47:47:0001
11,53:005
10.47:0000
12:30:000
5:005
18 08.07:10000
10.10:1:0000
10.73:10.011
10.23 0.00
10.97-1:.02
17.57
8=0
19.0 09
12.30 04
1087:20011
12.83=000 0.06
11:97:0004
19.91
A-03
00000000
11.7744022
10.03:20001
0.00
10.97
CIFAR-10
6:03
24 0.22
11:07:::000
11.77:10002
10.9700.00
11.33:10000
00:07:1::33
8=10
29 10:00000
12.00:1::00
11 00:00000
13.23=0.00
136120011
28:23
AC-1
10 00
10. 000000000
10.27:
10.23-0::0
10.37=0001
10.00:00000
1897
01,10::001
00:00001
11.53:0011
10.77:0000
14.77:0011
4-001
10
15.53+0.30
8.870001
0.00
9.33+0000
15:00+0244
60100.1
07:00
77:42:1144
47.73:20222
07.40:0000
00
4-03
002
67 00
67 00.07::0:115
66.17=0.21
63. 0..40:00.11
84 00:0000
MNIST
8=0.5
88 0.7
79.27.0000
78, 57.
77,37-000 0.07
70 00
86. 06.33:0000 36
(-10
01
84
01:40:10.01
00:27:0000
10.10:10.011
10,10:0001
10:13:10:01
9
AC=2
60 63:22
21 50:000040
25 23:
16.47-0.23
14 04.30.20.30
$2.73:1:466
#C-3
77.132033
29.53 1K5
31 234
33.13=2661 2.60
20:00::2.04
08.00:0.31
A-005
22.93:2018
14 04:00:444
15:77:20144
16.60-0.21
15.00:10
21 21
[H-11
09.77=000
13
25.70:20008
22,17 17-0:2
0..50:20.06
19.43:0455
8-0.3
02223:0026
34.40-20226
00.0
0.26
34.70::0.20
47.13+7.14
0427200022
07
10
38.51-0.15
03.70:20007
3-1.0
67 0.000:00011
33.00:008
04
55.97-0.04
55.23:2.112
54 54.40119.45
#C=l
19:0010.00
00.43::000
13.73:0018
13.77-0.17 0.17
18.27-00.03
7,70:::003
47.0311433
12.00:0.20
17.47.0.13
37.6710766
208740.12
07:00000
27.60-0003
2493-00110
47.43+0.40
Training Details. By default, we follow FedAvg [12 and other existing studies 667 68 15
to
use simple CNN with layers in our experiments. The experiments with more complex neural network structures in Appendix G.8 We set the batch size to 64, the learning rate to 0.001, and the À 0.001 for FedLPA. By default, we set 10 clients and run 200 local epochs for each client. For the various settings of the number of clients and local epochs, we refer Section 4.3 and Section 4.5 For results with error bars, we run three experiments with 5 different random seeds. Note that all methods evaluated under fair comparison settings. Due to the page limit, representative results are represented in the main paper. Refer to Appendix G for more experimental details and additional results.
Baselines. To ensure fair comparisons, we neglect the comparison with methods that require to download auxiliary models or datasets, such as FedBE [69], FedKT [14] and FedGen [21], even pretrained large model, like FedDISC [20]. FedOV [5]] and FEDCAVE (23 entail sharing more client-side label information jeopardize label privacy and are beyond the scope this study. XorMixFL [22 may not be practical,
transmitting client label information to the server, which could
mentioned before. FedFisher 41 is not publicly available. FedDF (26). DAFL [35] and
ADI (3)] are compared with the state-of-the-art data-free method DENSE 24). Co-Boosting [39
requires too many computational resources In conclusion, we include one-shot FL algorithms as baselines including FedAvg [12), FedProx [31], FedNova [30], SCAFFOLD (2) and DENSE (24).
All the methods fairly compared, and our implementation available and the experiment details
can be viewed in Appendix G.1l
4.2 An overall comparison
We compare the accuracy between FedLPA and the other baselines as shown in Table the data in the green shadow shows the best results. FedLPA can achieve the best performance in all the dataset and partition settings. In extreme cases such significant performance advantage over the baseline algorithms. This demonstrates our framework's ability to effectively aggregate valuable information from local clients for global weight training.
[0.01, 0.05), #C 1, #C 2, FedLPA exhibits
The experiments with models, FedOV and Co- Co-Boosting, are in Appendix G.5

---

Table 2: Experimental results of varying number of clients FMNIST dataset.
of Clients
Faniton
FeoLFX
FedNova
SCAFFOLD
FedAvg
Foul
DENSE
8:0.1
33.5720.35
10:00:0000
13,1320.34
13.2320.311
13.9=0.05
10. 00:00:0000
21,20:1:008
21.20:10.61
06.13-1:0.36
8:0.1
57.37:10.05
31,5011::299
29.23=00.00
32.43:10.99
28 1..00-1..2
57.20:20011
8:0.3
71.3010.00
53.8720.31
50.6320.10
52.83220008
52.13=0.40
71.174:004
Clients
8-0.5
14,07::0.00
62.83:1:000
58.6010.008
60.17:10.03
59,47 0.06
74.00:10.01
#=1.0
686310008
69.13:.0112
68.13:10.08
75.41:0.04
21.5010.30
10:00:0000
10.00=0.00
10:00:00000
10.33=0.00
10.00::0000
4C-2
59.17:10..5
1922311:223
19,97
18.53:100:6
13.53 0.26
33.07:1:0.27
4C=5
66.37100.01
27.30:1::00
28.07
25.93:10.27
24.63=0.20 0.26
52.21-1:079
0.01
15.9120.01
10.0000000
10.00000000
10:00:00000
10.00+0.000
8:0.05
28.43:10080
15.50:1:0443
17.77=00.25
17.37.10.24
18. 10-00-0:000 0.01
25.03
8=0.1
$7.0 07.08:20.00
34.33:1:004
00.17=0..3
28, 0.05
31 31.00=022 0.27
55, 1..---200000 49
9-0.1
26.20:2:1223
46.70:3.12
43.97=0.02
05.1122..12
59.23-4:100
Clients
8:0.5
11.13:1:.000
57.93:1:040
52.93::0.22
53.67:10.066
03.80=0.20
69.57:10.02
8=1.0
71 11.07:00.044
6000200000 0.00
$7.67=0.222
56 10:12.04445
56.90 16.90=0.41
70.33:20000
SC-1
15. 15.9340.002
10:00::000
10.00=00000
10.00:00000
10:27=0:00
10,00:0.00
4C=2
19.60:10.37
180010.111
17.20:0.000
0.220
44,571:0.92
65.90:00.05
38.03:2:009
40.53=1.411
40.97:1
38.93=1
10::0.3
In summary, the state-of-the-art DENSE could be comparable with FedLPA when the skew level is small. However, with the increment skewness, FedLPA shows significantly superior results.
4.3 Scalability
We assess the scalability of FedLPA by varying the number of clients. In this section, show results on FMNIST in T From the table, we can observe that FedLPA still almost always achieves the best accuracy when increasing the number of clients. Notably, there is slight exception highlighted in red, where DENSE outperforms us when we have 20 clients and 3 = 0.5, this may be attributed to the dataset being less biased and the DENSE only getting marginal 0.03% higher test accuracy. Our method is generally much more robust in all kinds of settings.
Table 3: Experiments with different proportions of data samples. Data sample proportion
Accuracy(3=0.1
Accuracy(1=0.3)
Accuracy(3=033)
100%
55,33=0.06
68.20=0.04
73.33=0:06
80%
53.88+1.14
65.47+0.02
73.17+0.005
60%
53.15=0.82
61.8020.71
72.40=0.29
40%
53.20=0.21
64.10=0.40
70.02+0.17
20%
45.71=0.13
62.15
68.54+2202
4.4 Experiments with different proportions of data samples
We have added the experiments with our method on the same experiment setting with 10 clients. We conducted experiments on FMNIST datasets with 330.11 0.3 and 0.5. The performance changes w.r.t the number of data samples are shown in Table We could see that method FedLPA could yield satisfactory results even with only 20% data samples under multiple settings.
4.5 Ablation study
The hyper- paper-parameter of our approach is X, which controls variances of a priori normal distribution and guarantees A and B₃ are positive semi-de semi-tefinite. In this part, we show results on FMNIST. All other Laplace Approximations are sensitive to the hyper-paramete based on their experimental results, Table 4hoows shows that our approach is relatively robust. Based on our numerical results, we set 0.001 by default for our method FedLPA.
We also conduct the experiments when the local epochs are 10,20,50, 100. More experiments are available in Appendix G.2 which shows that our methods outperform all the baselines in all kinds of scenarios without requiring extensive tuning.
4.6 Communication and computation overhead
We conduct experiments on CIFAR-10 on single 2080Ti GPU to estimate the overall communication and computation overhead. We set the number of clients is 10. Table shows the numerical results on

---

Table 4: Experimental results of different hyper- pyper-parameter on FMNIST dataset.
Table 5: Communication and computation
overhead evaluation.
0.00
0.0001
3=0.01
0..78
21.20.10.77
22.50.11.4
Computation (mins)
Communication (MB)
3.0.05
54.33:1..54
54.27:10.38
53.30:10.01
FedL.PA
4,98
0=0.1
19
55.33=0.06
54.60=0.15
FedNova
2.47
6=0.3
66.83-10.02
68.20:10.04
67.53-10.03
SCAFFOLD
50
4.94
0=0.5
73.20=00.03
73.33-#0.06
72.17=0.04
FedArg
6=10
76.53=0.02
76.03=20.05
73.47=0.19
FedProx
2.47
#C=1
12.73-10.01
13.20:10.02
14.17:10.02
DENSE
400
2.47
#C=2
45.20:10.21
46.13:10.15
04.00.10.03
#C=3
58.97=00.07
57.90=10.06
55 05.000
FedLPA and baselines. Details of the overhead evaluation referred to Appendix Gnd and G.7 Our observations reveal that FedLPA is slightly slower than FedNova, SCAFFOLD, FedAvg, and FedProx while much faster than DENSE. FedLPA also has significantly improved the one-shot learning performance of the above four approaches. Similarly, Fedl .PA performs moderately incremental communication overhead while outperforming other baseline approaches learning performance, as one-shot FL. introduces heavy computation overhead while communication overhead usually small. It is noteworthy that FedLPA strikes favorable balance between computation and communication overhead, making it the most promising approach for one-shot FL.
4.7 Extension to multiple rounds
We conduct experiments on MNIST with 10 clients and data partitioning Ps:- Dir(3 0.5). The results are shown in Figure 2 As DENSE could not support multiple rounds, we compare
Test Accuracy Curves
methods with Fe- d'vgg, FedNova, SCAFFOLD, and FedProx. FedLPA achieves the highest accuracy in the first round, denot- ing the strongest learning capabilities in a one-shot setting. With the increment the number of rounds, the performances of FedLPA increase slower than the other baseline approaches. This figure shows that the joint approach (ours (one round) then FedAvg) that utilizes FedLPP in the first round and then adopts other baseline methods may be most promising to save communication and computation resources in the multiple-round federated learning scenario.
0.95
Test Accurasy
0.90
0.80
0.75
0u1
0us Jan than todes
5
10
15
20
25
Epochs
Figure 2: Extension to multiple rounds on MNIST dataset.
4.8 Supplementary experiments
Experiments for privacy concerns, experiments on different local epoch numbers, experiments in extreme settings (the number of clients=5, 3 0.001), experiments with more methods, experiments with more complex network structures, experiments with more complex datasets, ablation experiments analyzing the number of approximation iterations of FedLPA can be found in Appendix.
5 Conclusions
In this work, we design novel one-shot FL algorithm FedLPA better model the global parameters in effective one-shot federated learning. We propose method that could aggregate the local clients in a layer-wise manner with their posterior approximation via block-diagonal empirical Fisher information matrices, which could effectively capture the accurate statistics of a locally biased dataset. Overall, FedLPA stands out as the most practical and efficient framework that conducts data-free one-shot FI., particularly well-suited for high data heterogeneity in various settings, considering it significantly outperforms other baselines with extensive experiments. Our Fedl PA available in https://github.com/// htwp://hww..oom/eeemommmno////ee Det.tee/AAAAAAPPPANSAAPP2200
NeurIPS2024.
10

---

Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers. The authors would like to thank Yiqun Diao from National Univeristy of Singapore for his valuable comments to improve this work. Dr. Jialin Li is supported by the Singapore Ministry of Education Academic Research Fund Tier 1 (TI 251RES2104) and Tier 2 (MOE-T2EP20222-00116
References
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(75533:436-444, 2015.
[2] Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85-117, 2015.
[3] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg. Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech
2: End-to-end speech recognition in english and mandarin. In International conference machine learning, pages 173-182. PMLR, 2016.
[4] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90 00, 2017.
[5] Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei- Ling Shyu, Shu-Ching Chen, and Sundaraja S lyengar. A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR), 51(5):1-36, 2018
[6] Qingchen Zhang, Laurence Yang, Zhikui Chen, and Peng Li. survey on deep learning for big data. Information Fusion, 42:146 12:146-157, 2018.
[7] Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei- Ling Shyu, Shu-Ching Chen, and Sundaraja S Iyengar. A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR), 51(5):1-36, 2018.
[8] Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 1092):1-9, 2019.
[9] H Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agūera Arcas. Federated learning of deep networks using model averaging. arXiv preprint 1602v 2, 2016.
[10] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2): 1 1200 2021.
[1]] Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering. 2021.
[12] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficle learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[13] Neel Guba, Ameet Talwalkar, and Virginia Smith. One-shot federated learning, arXiv preprint arXiv:1902.11133 2019.
[14] Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. arXiv preprint arXiv:2010.01017. 2020.
[15] Yiqun Diao, Qinbin Li, and Bingsheng He. Towards addressing label skews in one-shot federated learning. In The Eleventh International Conference on Learning Representations, 2023.
11

---

[16] Maruan Al-Shedivat, Jennifer Gillenwater, Eric Xing, and Afshin Rostamizadeh. Federated learning via posterior averaging: A new perspective and practical algorithms. arXiv preprint arXiv22010.05273, 2020.
[17] Liangxi Liu, Xi Jiang, Feng Zheng. Hong Chen, Guo-Jun Qi, Heng Huang, and Ling Shao. A bayesian federated learning framework with online laplace approximation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[18] Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint (iii:::000007999,, 2020.
[19] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Jianghe Xu, Shouhong Ding, and Chao Wu. A practical data-free approach to one-shot federated learning with heterogeneity. arXiv preprint arXiv:2112.123711 2021.
[20] Mingzhao Yang, Shangchao Su, Bin Li, and Xiangyang Xue. Exploring one-shot semi-supervised federated learning with pre-trained diffusion model. arXiv preprint arXiv:2305.04000 2023.
[21] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for het- erogeneous federated learning. In International conference on machine learning, pages 12878-12889, PMLR, 2021.
[22] MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Scong- Lyun Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning. arXiv preprint arXiv:2006.06 048, 2020.
[23] Clare Elizabeth Heinbaugh, Emilio Luz-Ricca, and Huajie Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2022.
[24] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: free-free one-shot federated learning. Advances Neural Information Processing Systems, 35:21414-21422 2022.
[25] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi- task learning. Advances in neural information processing systems, 30, 2017.
[26] Tao Lin, Lingjng Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2363 2020.
[27] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint (xXiv:1806000582) 2018.
[28) Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the conver- gence of fedavg on non-iid data. arXiv preprint 190iv:190..02//00 2019.
[29] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.
[30] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7622 2020.
[31] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 11449-4500 2020.
[32] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning neural networks. In International Conference on Machine Learning, pages 7252-7261. PMLR, 2019.
12

---

[33] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khaza- eni. Federated learning with matched averaging. arXiv preprint arXiv:2002.064440 2020.
[34] Romain Thibaux and Michael Jordan. Hierarchical beta processes and the indian buffet process. In Artificial intelligence and statistics, pages 564-571. PMLR, 2007.
[35] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE international conference on computer vision, pages 35 3514-3522, 2019.
[36] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8715-8724, 2020.
[37] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, pages 9489-9999 9502. PMLR, 2021.
[38] David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv:1710.04759, 2017.
[39] Rong Dai, Yonggang Zhang, Ang Li, Tongliang Liu, Xun Yang, and Bo Han. Enhanc- ing one-shot federated learning through data and ensemble co-boosting. arXiv preprint arXiv:2402.15071 2024.
[40] Shangchao Su, Bin Li, and Xiangyang Xue. One-shot federated learning without server-side training. Neural Networks, 164:203-21 215, 2023.
[41] Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedfisher: Leveraging fisher infor- mation for one-shot federated learning. International Conference on Artificial Intelligence and Statistics, pages 1612-1620. PMLR, 2024.
[42] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020.
[43] Hippolyt Ritter, Alcksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference Learning Representations, ICLR C018-Conference Track Proceedings, volume International Conference on Representation Learning, 2018.
[44] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortlesss bayesian deep learning. Advances in Neural Information Processing Systems,
2021.
[45] David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back-propagating errors. nature, 323(6088):533-553 1986.
[46] James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference machine learning, pages PMLR, 2015.
[47] Charles F Van Loan. The ubiquitous kronecker product. Journal of computational and applied mathematics, 123(1-2):85-100, 2000.
[48] James Martens and Roger Grosse. Optimizing neural networks with Cronecker-factordd approximate curvature. In International conference machine learning, pages PMLR, 2015.
[49] Roger Grosse and James Martens. A Cronecker-factored approximate fisher matrix for con- volution layers. In International Conference on Machine Learning, pages 573-582, PMLR, 2016.
13

---

[50] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)33521-33
2017.
[51] Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703-17716 2022.
[52] Frederik Benzing. Gradient descent on neurons and its link to approximate second-orde optimization. In International Conference Machine Learning, pages 1817-1853. PMLR, 2022.
[53] Lin Zhang, Shaohuai Shi, Wei Wang, and Bo Li. Scalable k-fac training for deep neural networks with distributed preconditioning. IEEE Transactions on Cloud Computing, 2022.
[54] James Martens. Second-order optimization for neural networks. University of Toronto (Canada), 2016.
[55] Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. In International Conference on Machine Learning, pages 557-565, PMLR,
2017.
[56] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-2665222 2844. Springer, 2006.
[57] Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and S Yu Philip. Privacy and robustness federated learning: Attacks and defenses. IEEE transactions neural networks and learning systems, 2022.
[58] Andrew C Yao. Protocols for secure computations. In 23rd annual symposium on foundations of computer science (sfcs 1982), pages 160-164. IEEE, 1982.
[59] Daniel Demmler, Thomas Schneider, and Michael Zohner. Aby-a framework for efficient mixed-protocol secure two-party computation. In Network and Distributed System Security Symposium, 2015.
[60] Taher ElGamal. A public key cryptosystem and a signature scheme based on discrete loga- rithms. IEEE transactions information theory, 31(4):469-472, 1985.
[61] Pascal Paillier. Public-key cryptosystems based on composite degree residuosity classes. In International conference on the theory and applications of cryptographic techniques, pages 223 233.238.77 Springer, 1999.
[62] Craig Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pages 169-178, 2009.
[63] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278 2222222222222 1998.
[64] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 110.v:/70..07/444 2017.
[65] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[66] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[67] Tianhao Wang, Johannes Rausch, Ce Zhang, Ruoxi Jia, and Dawn Song. A principled approach to data valuation for federated learning. Federated Learning: Privacy and Incentive, pages 153-167, 2020.
14

---

[68] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965-978. IEER, 2022.
[69] Hong-You You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv preprint arXiv:2009.01974, 2020.
[70] Roger Grosse and James Martens. A soonecker-factored approximate fisher matrix for con- volution layers. In International Conference on Machine Learning, pages 573-582. PMLR, 2016.
[71] Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss newton optimisation for deep learning. In International Conference on Machine Learning, pages 557-565, PMLR, 2017.
[72] Elaine Angelino, Matthew James Johnson, Ryan P Adams, et al. Patterns of scalable bayesian inference. Foundations and Trends@ Machine Learning, 9(2-3):1 19-247, 2016.
[73] Ward Cheney and David Kincaid. Numerical mathematics and computing. International Thomson Publishing, 1998.
[74] David A Belsley, Edwin Kuh, and Roy E Welsch. Regression diagnostics: Identifying influential data and sources of collinearity. John Wiley & Sons, 2005.
[75] M Hashem Pesaran. Time series and panel data econometrics. Oxford University Press, 2015.
[76] Lloyd N Trefethen and David Bau. Numerical linear algebra. SIAM, 2022.
[77] Jongseok Lee, Matthias Humt, Jianxiang Feng, and Rudolph Triebel. Estimating model uncertainty of neural networks in sparse information form. In International Conference on Machine Learning, pages 5702-5713. PMLR, 2020.
[78] Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent kronecker factored eigenbasis. Advances Neural Information Processing Systems, 31, 2018.
[79] Jiankai Sun, Xin Yang, Yuanshun Yao, and Chong Wang. Label leakage and protection from forward embedding in vertical federated learning. arXiv preprint arXiv:2203.01451, 2022.
[80] Aidmar Wainakh, Fabrizio Ventola, Till Müßig, Jens Keim, Carlos Garcia Cordero, Ephraim Zimmer, Tim Grube, Kristian Kersting, and Max Mühlhäuser. User-level label leakage from gradients in federated learning. Proceedings Privacy Enhancing Technologies, 2022(2):227- 244, 2022.
[81] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage collaborative learning. In 2019 IEEE symposium on security and privacy (SP), pages 691-706. IEEE, 2019.
[82] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning? Advances in Neural In- formation Processing Systems, 33: 16937-1777977, 2020.
[83] Cynthia Dwork. Differential privacy. Encyclopedia of Cryptography and Security, pages
338 188-340,20 2011.
[84] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9(3-4):211-407 2014.
[85] Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557. 2017.
[86] Yiqun Diao, Qinbin Li, and Bingsheng He. Exploiting label skews federated learning with model concatenation. arXiv preprint arXiv:2312.062000 2023.
15

---

[87] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference computer and communications security, pages 1322-1333, 2015.
[88] Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz. Deep models under the gan: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC conference computer and communications security, pages 603-618, 2017.
[89] Cynthia Dwork, Adam Smith, Thomas Steinke, Jonathan Ullman, and Salil Vadhan. Robust traceability from trace amounts. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 650-669. IEEE, 2015.
[90] Apostolos Pyrgelis, Carmela Troncoso, and Emiliano De Cristofaro. Knock knock, who's there? membership inference on aggregate location data. arXiv preprint 1rXiv:1708.06145, 2017.
[91] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium security and privacy (SP), pages 3-18. IEEE, 2017.
[92] Jamie Hayes, Luca Mclis, George Danezis, and E LOGAN De Cristofaro. Membership inference attacks against generative models. URL htsps://api. semanticscholar. org/CorpusII), 202588705, 2018.
[93] Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. Understanding membership inferences on well-generalized learning models, arXiv preprint arXiv:1802.04889 2018.
[94] Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Towards demystifying membership inference attacks. arXiv preprint arXiv: 1807.09/73, 2018
[95] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[96] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[97] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNV), pages 2921-2926. IEEE, 2017.
[98] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
[99] Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
[100] James Martens, Jimmy Ba, and Matt Johnson. Kronecker-factored curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018
[101] Blair Bilodeau, Yanbo Tang, and Alex Stringer. On the tightness of the laplace approximation for statistical inference. Statistics & Probability Letters, 198: 10989, 2023.
16

---

A The FedLPA algorithm
The proposed algorithm follows the same paradigm as the standard one-shot federated learning framework. Each client follows the local training procedure shown in the paper. The global aggregation illustrated in Algorithm
With the Algorithms, let us assume the dimensionality list of each layer in a fully connected neural network is ([5g, $1, $2, St..... SLD, which means the size of the weight Wk, of layer I is St-1X8|. Consequently, the size of Ak, for this layer would be $1-18$1-1, and the size of BR, would be s(XS). The size of FA, is (81-128))x(5))113))
Then, we give a concrete example to show the dimensions of different matrices using a fully- connected neural network model with architecture 784-256-64-10 10 as in Appendix [ Then, the Mr, is 784x256+256, MR, is 256x64+64, MAg is 64x10+10. The Ak, is 785x785, Ak, is 257x257, As, is 65x65. The BR, 256x256, BR, is 64x64, BNg 10x10. Then the FR, is (785x785)x(56x2566) FA, is (257x25))x Fkg is (65x65)x(10x10)) The F& is
However, in fact, we do not need to combine the Ak.. B FN, into Ax, BR, Fg. In this paper, we utilize the diagonal block property to compute each block in our method.
B Comparison with the previous methods
To the best of our knowledge, we are the first consider the posterior inference problem in the one-shot scenario. Note that the approach [16] requires a lengthy burn-in period before conducting posterior inference, for instance, 400 rounds, and it updates global model parameters by modifying the wwvarianceaaggegaan
[16] necessarily requires multiple iterations and cannot be used in one-shot scenario. In contrast, our method FedLPA only requires immediate variational inference after training the local model, ensuring higher flexibility and efficiency in the one-shot scenario.
local models. It means that the algorithm
Besides, in the algorithm [16), obtaining statistical information to compute local covariances is of low rank. In reality, it fails to acquire the posterior of the aggregated model and cannot perform variational inference on the aggregated model. However, our method yields full-rank covariances, and after employing an expectation approximation method for variational inference the aggregated model, we can achieve a usable global posterior.
70 16 and modeling output uncertainty in variational inference (71), using the Fisher approximation of the Hessian does not involve the issue of inverting covariance. However, in the context of federated learning, when performing variational inference on the aggregated model, the necessity of inverting covariance becomes unavoidable. To address this problem, we propose a novel algorithm that constructs a quadratic objective function. During aggregation, this algorithm directly trains the aggregated model using local covariances and expectations, thereby circumventing the need for inversion operations.
In both the domain of natural gradient optimization 48
Note that the previous methods [51 17 adopt the same core approach that utilizes the online Laplace approximation to obtain diagonal Fisher for model aggregation, in which they conduct experiments on different datasets and published on different venues. We mainly analyze our approach with the comparison of DisgonalFisher [17. DiagonalFisher assumes independence among parameters, neglecting mnter-parameter correlations, resulting in inaccurate posterior approximations. However, strong correlations exist among parameters within each layer, such as matching patterns in convo- lutional kernels within convolutional networks. This is a crucial factor that cannot be overlooked; otherwise, aggregation of the posterior would result in lower posterior regions, as compared to method. In complex environments, employing diagonal Fisher for aggregation would prove to be entirely ineffectivve whereas our method effectively leverages inter-parameter correlations at each layer, rendering it more robust. To demonstrate, we present results comparing aggregation using diagonal Fisher and our method. We have added experiments using the settings of our paper and an MLP model (784-256-44 10) on the FMNIST dataset with five random seeds for one-shot FL, the client number is 10, and the 3=0.01. The results are in Table
In the table, "Initial" denotes whether the client models were initialized using the same parameter values or independently.
17

---

Table 6: Experiments with DiagonalFisher using MLP.
Initial
FedAvg
FedProx
SCAFFOLD
DiagonalFisher
FedLPA
Same
4.. 10.00+0..00
24.80+F..10 24.12+00.02
10.105
56.34
76.63+00.04
Different
10.16+0..70
10.51+0.11
73.73+0007
When "Initial" is set "Same", all client models are trained on their respective datasets using identical parameter values for initialization. Consequently, there exists strong correlation among the local models. Additionally, this scenario, model aggregation equivalent aggregating updates of local models. Although DiagonalFisher performs reasonably well under this condition, our method demonstrates superior performance, exhibiting a 20.29% increase in global test accuracy.
When "Initial" is set to "Different", the models on different clients start training with distinct parameter values. Due the high heterogeneity of local datasets, there is minimal correlation among local models. In this extreme scenario, DiagonalFisher completely fails, while our method maintains an accuracy of 73.73%, showcasing remarkable robustness.
is essential to consider the indispensability of parameter correlations, which is why we com- pute correlations among parameters within layers ensure the robustness and accuracy of model aggregation.
Now, we discuss some related works which directly utilize K-FAC to approximate the Fisher matrix and make a comparison with our proposed approach FedLPA. The works [48] 70 71 have provided us with significant inspiration. However, methods like K-FAC do not require computing the inverse of covariance. Nevertheless, in the context of federated learning, the necessity of inverting covariance becomes unavoidable during variational inference on the aggregated model.
Methods like K-FAC assume direct independence among data samples utilize expectation approx- imation. They obtain the inverse of Fisher from individual samples and then directly compute the expectation, thereby avoiding inverse operations. However, the expectation approximation inevitably leads to biased results during model aggregation. Detailed analysis can be found in Appendix [
To address this issue, we propose novel algorithm that constructs quadratic objective function. During aggregation, this algorithm directly trains the aggregated model using local covariances and expectations, eliminating the need for inversion operations. This aims minimize aggregation biases as much as possible.
Here, provide comparative analysis of different methods.
FedAvg and FedProx minimize the Kullback-Leible (KI.) divergence between the local and global posteriors: μ. Σ min KL (EE K p(0ID)))p((ID) SCAFFOLD computes the bias term, and FedNova computes the correction term. None of these four methods consider the corre- lations between parameters. DENSE and FedOV, on the other hand, employ distillation methods, attempting to extract the distribution of non-iid data among clients through distillation. However, this itself leads to information loss due dimensionality reduction and introduces additional variance of data.
Although the work [72] also uses the distributed Bayesian inference, however, it focuses the dataset feature and could not be applied to train the global model parameters.
In conclusion, the reason our approach performs better in this scenario stems from our improved approximation of the global posterior. This approach signifies our novelty in addressing these challenges.
B.1 The efficiency of FedLPA
Although the number of uploaded bits increased per round of FedLPA, it resulted in a significant improvement in the final outcome. Additionally, the increase in transmitted bits enhanced the robustness of the aggregation method. Moreover, as indicated in Table 3f the paper, we observe only marginal increase in the amount of communication required.
fully-connected neural network model with architecture 784-256-64-10 10, has 784 256 + 256 256 64 + 64 + 64 10 + 10 217930 floating point numbers, which is 6973760 bits or around 0.831
18

---

Table 7: Experiments with DiagonalFisher considering efficiency.
"Initial" Method
FedL.P (Global Test Acc / MB) Diagonal (Global Test Acc/ MB)
"Same"
16.63/(2.272 3.37
56.34/( 16622 10) 3.39
"Different"
73.73/(2.222 10) =325
(1.61/11.622111 = =0.3
MB. For one communication from a client to the server, our approach needs to upload additional A and Bg, which have 785 785 + 256 256 + 257 257 + 64 64 + 65 65 + 10 10 = 756231 floating point numbers. Note, As and Bg are symmetric matrices, so we only need to upload the upper triangular part of A₁ and B; which is around 75623 = 3781 778115. 15.5 floating point numbers and 1.442 MB. Therefore, our approach costs 2.272 MB for the directed communication. which only 1.367 times than DiagonalFisseer while DiagonalFisher costs 0.831 *2 = 1.662 MB. We show the following Table]) based on the previous experiment results. When "Initial" is set to "Same", the efficiency of every bit is almost the same. When "Initial" is set to "Different", the efficiency of every bit for our method is much higher than the DiagonalFisher.
C Detailed discussion for the time complexity of Eq. 12
A fully-connected neural network model with architecture 784-256-64-10 as an example is shown in Appendix K. We use this example to further explain this question. The size of Ak is 785x785 and the size of BR, are both 256x256. Then, we need to compute the inverse of the matrix (785x785)(256x222
which is huge. The time complexity of calculating the inverse of a ma- trix is (n²²) (n is the dimension of the matrix), which is very slow. The accuracy of calculating it is decided by the condition number of the huge matrix 773 is unacceptable, considering the time complexity, the size of the huge matrix and the accuracy.
76). That's why calculating Eq. [2
Further, for example, in the machine learning field, accelerate the training of the neural network, they use the Newton method. However, using this method, they need to compute the inverse of the Hessian matrix, which is also huge and unacceptable. That is why they introduce the KFAC 46 49)
KFRA (55) and KFLR (55) methods to avoid computing the inverse of the huge Hessian matrix.
In this paper, avoid computing the inverse of the huge matrix via method, and the time complexity is linear.
D Expectation approximation (EA)
Previous works 46
49 approximate the expectation of Kronecker products by a Kronecker product
of expectations E
A
B E [A] @ E [ with assumption of A and B₁, are independent,
which is called Expectation Approximation (EA).
It is simple and effective method to approximate the expectation of Kronecker products. As result, the global co-variance Σ is approximated by:
K
K B B,
Σ
A
(15)
where A, = ΣK Ag, and B Σσ B Denoting Z, as matrix formula of 2, = vec(Z,), then j. can be computed efficiently as below:
P Σ. 2 8A "
(16)
However, Eq. 16 leads to biased global expectation. The EA needs the independence assumption, but A1, and BR, are weakly related back pock-propagation. Besides, even if they are independent, Eq. 16 still suffers from approximation error because the clients' number K is finite and always small number but statistical independence can only be demonstrated when the sampling number is large enough. Eq. [7 the approximation error directly:
(A1 + A2) & B + B2) = Ai
X
B + A & B2
+A 2 B + & B
(17)
# A
B + A 8 B2
19

---

E
Extend FedLPA to the models with enormous single layer weight parameters
This implies a Fisher matrix with a large dimension and it significantly increases communication costs. In such cases, the most intuitive approach is to explore the possibility of dimensionality reduction for its Fisher matrix. A promising approach to enhance the efficiency of our method may employ some low-rank factorization techniques (77]. As described [44], the main idea involves perform- ing an cigendecomposition on the Kronecker factors [78), while preserving only the eigenvectors corresponding to the top k largest eigenvalues, As a result, this approach drastically reduces space complexity, enabling communication costs to be compared favorably with diagonal Fisher matrices.
F Privacay discussion of FedLPA
In the FedLPA, Aş is computed via the activations while B3 is computed via the linear pre-activations of the layer. We note that A₁, Bx, and M2 do not carry any label information, thus the transmission of Ak. Bg, and Mr will not leak any label privacy. As a comparison, FedCAVE, which transmits client label information to the server, requires training in label distribution to do the distillation. Several papers 779 80) have notified that label privacy, e.g., the concern of label distribution leakage and raw label leakage, is sensitive in federated learning. We believe that it has also been concern in the one-shot FL scenario.
Besides, our t-SNE illustration in Fig T shows the classification capability on the global model, which can separate the classes. However, our figures of the t-SNE illustrations on local models in Appendix G. show that for the data belonging to the same class, their t-SNE illustrations are erratically distributed on different local nodes. For instance, for node 2, its training data only has classes while most of the training data locates in class 5. However, it is hard for the server to infer that label distribution since the t-SNE illustration both on node 2 and other nodes also seems irregular.
Ag. Bg, and Mj are a function of data that may contain privacy-sensitive information of the local training data. However, in this case, our grivacy-preservve that FedLPA has the same srivacy-preeervngg level as the conventional federated learning algorithms (i.e, FedAvg. FedProx, FedNova, and Dense), which are all vulnerable some privacy attacks (c.g, membership inference [T) or reconstruction attacks (2)). Our approach FedL.P provides more information than FedAvg, However, the additional information we provide is the mean of each sample in each dimension, the mean of squares of each sample in each dimension, and the mean of square gradients. These solely marginally enrich the attack capability of several reconstruction attacks.
level is similar to FedAvg, which means
FedLPA is intuitively compatible with existing privacy (DP)
preeeevine techniques, such as differential [56 577] secure multiparty computation (SMC) [58] (9), and homomorphic encryption
(HE) 660 61 62 In Appendix F.T we propose a naive DP-FedLPA with two different mechanisms to show the compatibility with differential privacy. In Appendix F22 using iDLG attack (82), show that our proposed FedLPA has the same privacy-preserving level as the conventional federated learning algorithms (i.e, FedAvg. FedProx, FedNova and Dense). Compared with FedAvg, we have conducted a detailed analysis from a privacy attack perspective to show that our proposed FedLAP exhibits security level consistent with FedAvg against several types of privacy attacks, where the details are shown in Appendix[F3 FF3 Note that the main focus of FedLPA is to improve the learning performance on the one-shot FL settings, thus, we leave the integration with other privacy-preserving techniques beyond DP as an open problem.
F1 Experiments with differential privacy
We first list the definitions and techniques for differential privacy 83 (c-DP For > O, a randomized function f provides c-differential privacy for any datasets D, D' that have only one single record different, for any possible output O
Pr(f(D) E 5 -r(f(D') E O
(18)
Suppose f is a function and D, D' have only one record different. The sensitivity of is defined as [l - f(D'lla
Af = max
(19)
D,D
20

---

Table 8: Experiments with Differential Privacy using two mechanisms,
Partitions
FedAvg
DP-Fedl PA (mechanism 1) DP-FedI.PA (mechanism 2)
3=0.1
31.90+H0.58
50.01+0.077
57.15++.223
8
8=0.3
44.37=10.05
68.30:0.41
66.21
0=055
57.92-+-0.63
71.17+00.77
73.50-+0.06
8=0.1
28.1710.16
48.5110.07
55.87 0.88
5
8=0.3
43.91110.00
67.34-10.92
66.02:10.71
3=0.5
57.14:10.63
70.89+0.80
73.44+0.20
3=0.1
27.85++0.79
48.39:00077
54.31
3
8=0.3
42.80+10.05
65.08
65.22++0.46
$=0.5
54.80-10.63
70.2811.30
12.19
Table Experiments with Differential Privacy for Round Numbers.
B
8
0.1
11
11
12
0.3
11
9
0.5
8
8
Here record different means database has more record than another. We utilize the Laplace mechanism [84] to achieve the e e-DP.
Laplace Mechanism: For function f ; > Rd function:
F(D) f(D) + Lap(0, Af/c)
(20)
provides e-DP, where Lap(O, Ae) is sampled from Laplace distribution.
86 to protect privacy, we conduct the two mechanisms of DP-FedLPA: (1) adding Laplace random noise the training data samples, (2) adding Laplace random noise to the parameters to be transmitted. DP is rigorous and popular privacy metric, which guarantees that the output does not change with high probability even though an input data record changes. Specifically, since the sensitivity of the data sample distribution after the normalization
Following the differential privacy (DP) mechanisms [56
85 57
We c (3. 5, 8) that provides modest privacy guarantees since normally E (1, 10) is viewed as suitable choice. We have added the experiments using the same experiment setting in the paper with five random seeds and 10 clients on the FMNIST dataset. Results are shown in Table g DP-FedLPA under both mechanisms outperforms FedAvg, which shows that is compatible with combining our proposed FedLPA with DP to enhance privacy protection levels. Note that the smaller is, the larger noises we add. We find that when the gets smaller, the performance drops simultaneously, while the privacy protection level is increased.
add Laplacian noises with
Besides, we have added the experiments using the same experiment setting to show the round results of how many rounds De-FedAvg needs achieve the same test performance with the first mechanism. The results in Table Table that DP-FedAAg needs about 10 rounds of communication to achieve the same test performance, compared to our one-round FedLPA. Combined with our previous results in Tablee5land Table our FedLPA could save the communication and computation overhead and combine with the DP method to mitigate the potential privacy leakage. Based on the above settings, De-FedAvg needs at least 3x communication overhead and 5x computation overhead. While DP- FedAvg needs multiple rounds to get similar accuracy, DP-FedAvg maybe vulnerable to more privacy attack methods due the multiple queries, such curvature- based privacy attacks.
F.2 Experiments with iDLG attack
We also add experiments with iDLG attack htttss//githb.coo/Paatte//Z com/Parrcc22///mmroeed Deep-Leakage-froet hraaaggeerroooaaaiieesssooobaaae We did the experiments with the setting of the paper [82]: in each single experiment, the client trained with one random picked image in FMNIST, then we use the iDLG attack to recover the image based on the model from FedAvg and FedL_PA We randomly selected 500 training examples to
82
following the link
collect 500 MSEs between the recovered and the original image. The larger the MSE is, the better
21

---

Table 10: iDLG attack results of Pe and FedAvg.
Percentile
12.5
25.0
37.5
50.0
62.5
75.0
87.5
100.0
FedLPA
0.60
1.00
1.10
1.39
2.75
50.71
40736.88
>=1e9
FodAvg
0.09
0.16
0.71
1,56
26.24
950.89
54307.91
>=1e9
level for the method. Due to the rebuttal limitation, we cannot show the figure for the cumulative distribution function considering the MSE of the iDLG attack. We provide the results in Table TO to show MSE considering the percentile for these 500 experiments.
the
Based on the Table, we could see that from 12.5 to 50.0 percentile, regarding the privacy-preserving aspect, FedLPA behaves better than FedAvg on these samples. However, from 50.0 to 87.5 percentile, FedAvg behaves better than FedLPA performs better when referring to the privacy level. Considering the overall 500 data samples, we roughly concluded that FedLPA and FedAvg share similar privacy level.
such samples. Thus, clear evidence exists of which one
F33 Concrete examples of privacy attack
For privacy attacks, we start by assuming the simplest scenario where each client has only one sample, and the model comprises a single layer, such as a multi-layer perceptron.
Let = W*x, *x, (x is n dimensional, with the last dimension being unit value 1), Df (where f is the loss function). In this case,
xx" B =g"".
In this single-sample scenario, an attacker can directly obtain x from the last column of A. With and W, the attacker can acquire the model's output. Furthermore, utilizing the Loss and g, it's possible to get the label information.
FedAvg would also be vulnerable to reconstruction attack in this scenario, allowing the attacker to obtain sample and label information.
When each client has two samples (x|X² E Dataset), then: A = 1/2=x|x=
B
T 1/2* *gg g1 * gf + 1/2 g2 gg The last column e of equals 1/2x1 1/2xg. The diagonal elements d of A equal 1 1/2x} 1/2x3. In the case of these two samples, an attacker can utilize the information from A and B to get the two samples X1 and x2. Using the same methodology, they can also obtain gi and g2 Consequently, the attacker can everse-engineer the labels as well.
FedAvg could also potentially succumb reconstruction attack in this scenario, providing the attacker with sample and label information, although the obtained information might be more ambiguous.
When each client has three or more samples (x E Dataset), A Ex Exx * xxx" B. Exe Dataset (g g)). In this situation, the last column of A, Ex Dataset (x) represents the average of the sample dataset, depicting the projection the data distribution in the sample space on various coordinate axes. Furthermore, the diagonal elements of A (Exe Dataset (x x offer the attacker statistical information about this local dataset.
Generally, solely using the statistical information of these datasets cannot reconstruct the entire dataset. Similarly, it's not possible to obtain gradients for the output of each sample, thereby preventing the reconstruction of individual sample labels. The results obtained by using and W to gather statistical label information are unreliable.
Additionally, for structures such as CNNs and RNNs/LSTMss, the difficulty of attacks increases due to weight sharing. For CNNs, since convolutional kernels only accept local samples input, information in A encompasses statistical information from all localities of the samples. For RNNs/LSTMSs information in A includes statisties of each word vector sentence. These network structures make it possible for attackers to fail even in single-sample scenarios. For MLPs, the information contained in the intermediate layer A is almost equivalent to the information encoded in the parameters of the BN (Batch Normalization) layer. The mean output of the Batch Normalization (BN) layer is equivalent to the last column of A, whereas the variances differ between the BN layer and A Ass diagonal but both contain statistical information related to squared values.
22

---

It's worth noting that the parameters acquired by the BN layer using the sliding-window method are also frequently used during the computation of A and B, mentioned in the paper [54].
window average
FedAvg provides model parameter values, the average of gradients, and BN layer parameters. Compared to FedAvg, the additional information offer is actually limited to: the mean of each sample in each dimension, the mean of squares of each sample in each dimension, and the mean of square gradients. Utilizing this information, attacking becomes highly challenging when the number of samples exceeds three, Although we don't It rule out the possibility of successful methods in practice due the data's ss own correlations, the limitations are significant based on our analysis, and our security level is quite close to that of FedAvg.
We discuss two common attacks here. Inferring class representatives:
Model inversion attacks 87 exploit the confidence information provided by machine learning applications or services. Our method does not provide confidence information, nor does it compute the information required for it. Therefore, our method's defense level against these attacks aligns with FedAvg's defense level.
ii) Attacks using GANs construct class representatives 88 utilize the client-uploaded model as discriminator and its output as labels to train generator to generate similar data. The additional statistical information we provide might be used constrain the distribution of inputs for GANs, specifically their mean values. Since the statistical information of the dataset may contain some common features among samples, might potentially aid in speeding up the convergence of training GANs but may not significantly enhance the accuracy of generated data after GAN optimization. It's worth noting that if the BN layer parameters uploaded by FedAvg could be used to constrain the statistical information of GANs' inputs, they would be equivalent to the information provided by our method.
Additionally, these attack methods against FedAvg only yield favorable results when class members are similar, meaning the dataset has clear common features that allow the constructed representatives to resemble the training data. When class members are dissimilar, these shared features tend to be confounded, rendering the constraints imposed by the sample mean ineffective, hence not enhancing the effectiveness of GANs attacks.
In summary, our method exhibits security level consistent with FedAvg against these types of attacks. Even in cases where the BN layer is not required, our method's security is similar to that of FedAvg.
Membership inference attacks against aggregate statistics 87 88] and Membership inference attacks against ML models [89] 90, 91 92 93 94 aim to infer whether a sample belongs to the training dataset using appropriate prior distributions and statistical data. These attack methods impose specific requirements on the dataset. In such attack scenarios, whether the sample mean information method can provide is exploitable by the attacker depends on whether this information can reveal the inherent distribution correlations within the dataset. However, for high-dimensional complex data, sample mean information often falls short in achieving this.
The inference attack towards client model is a complex topic. Other inference attack methods and defense mechanisms against them fall outside this paper's scope. is an interesting topic to explore more robust measures to prevent such breaches in future works.
Therefore, in the case of these attacks we mentioned, our method exhibits the same level of security as FedAvg (since FedAvg requires uploading statistically equivalent information within the BN layer). For scenarios without a BN layer, whether our method reduces security depends on the characteristies of the dataset itself. Real-world data is often high- ligh-dimensional and complex, making successful attacks challenging.
G Additional experiments
G.1 t-SNE visualization
We conduct experiments using MNIST dataset with a 3 value of 0.05, training 10 local clients over 200 local epochs with random seed O. In this biased local dataset setting, local clients could only distinguish a subset of the classes, illustrated in Figure
23

---

-an
-90
(a) local client
(b) local client
(c) local client #3
(d) local client
(e) local client
(f) local client #6
(g) local client #7
(h) local client #8
(i) local client #9
(j) local client #10
Figure 3: I-SNE visualizations of 10 local clients.
(a) FedAvg
(b) FodNova
(c) SCAFFOLD
(d) FedProx
(c) DENSE
Figure 4: t-SNE visualizations of the baseline approaches on the global model.
Based on seed 0, we partition the training data for the 10 local clients with the following form (label:# of the data)
local client [4: 2; 12, 2847, 9: 16)
local client #2: [1: 20, 4: 189, 5: 5349]
local client #3: [0: 669, 476, 2: 67, 15. 7: 6068)
local client #4: (0: 266, 1: 375,, 3956, 7: 196, 9: 5932)
local client #5: [0: 4, 1: 418, 5862)
local client #6: (1: 2, 25, 5195, 5: 24, 6: 80, 8: 28)
local client #7: [1: 5034, 2034,2:3, 3,4: 22, 6, 2669)
local client #8: (0: 4914]
local client #9: (4: 433, 5: 29, 307, 8: 5373)
local client #10: (0: 70, 1: 417, 2: 3: 2175, 4: 1; 1, 7, 8: 450,
It is worth noting that local client #2 has the training data mostly with label number 5, and as the corresponding t-SNE visualization shows in Figure 3b the local train model could mainly cluster the data with label 5 (marked as purple). As data for label (marked as orange) is different from other data with all other labels, some local clients may be able to cluster the data with label with good results. Other local clients, such as local client #3, #4, #6, #7, #9, #10, show the similar results like local client #2.
Figure 4 displays the t-SNE visualization for the global models of FedAvg, FedNova, SCAFFOLD, FedProx, and DENSE using the training data, with the figure legends identical to those in Figure It's evident from Figure that FedLPA outperforms the baselines in classifying the ten classes.

---

FedLPA's superiority is not only demonstrated by its ability to cluster the ten classes but also by the distinct separation between classes, as observed in Figure compared the baselines.
G.2 Experiments on different local epoch numbers
Table 11: Comparison with various FL algorithms in one round with 10 local epochs settings.
Dinaset
Fartiton
FedLFA
FedNova
SCARFOLD
FedAng
FedProx
DENSE
8-001
13.37
0-0.05
30.7749944
15.6740.28
20
19:5--0.30
20.07-0.37
23.00:0.14
040.1
42.3--0-33
25.30 1.32
21,03=100 1.08
22.57+-1J0 1:08
22:20-4-1:17
14.11-00.16
840.3
61.43:10.17
4..00:0.05
40,70:00.99
38.20:10.10
37.50:10000
43.17:10.05
2-0.5
01.77=0.46
3+1.0
71590+0009
51.06-00.22
52. 00.30-0.333
51.50-00.02
50504-0577
52.30--0.155
IC=1
13.00 1-004
10.00:00.02
10.90110.02
11.43
10:00100.000
90=2
28.93-0..74
16.34-0.19
0.01
25.3--00.00
23.33-4-008
22.60:5:038
22.80--0.40
82001
15.0100.0
10.0
0201
11901:000
11931000
100010000
040.05
2023+-0001
0050-+00002
10.00-0.000
10:00+00000
10:00-0000
14.34400004
8-033
25.601::001
18.13:1::333
0.99
14.77:10005
15.67:10000
20.33 1:000
CIFAR-10
9-0.5
25.00+0088
14.
16.77=0.011
15.73+-0001
13.92-+-0.8
25.20---0.16
3=10
28.011-0.01
15.63-1-.00
19,10:00,14
15.00
15.43-10005
22.30
10,23-0:00
10.23:1:0000
10=2
2000++000
11.37:00004
0.06
11.00+00000
11.00-20006
10.00-000000
KC-3
22 22.00-0-000
12:23 +0004
14.37
14.10:00.14
14.00-10.122
18.30-1-0.14
04005
6060+-0.07
20.30-00.13
0.66
35.10+0..77
34.13-1-991
50.37---.57
8-0.1
78.07 1:0.9
15.07 1:037
43.25
43.83-1:0.13
44.2711221
65 1.0.5
8:033
85 6010.117
64.10:1:344
64.03
64.17110.09
64.07.10.11
75.53-1:222
71.37=0.22
8=1.0
0..00
85.0011.110
85. 0.006
84.00:1:008
8163--00088
30:00000000
4C=1
11:07
10:43:1:000
10. 03.0
10.13:1:0001
10.13:1:0001
9.01
MC=2
47.93---.89
13.00-00.09
16.47 0.221
12.971-0.11
12:23-1-007
32 4:0.6
70:122224
2.66
8:001
1700100
13.0110.16
16:
16.27110:22
13 20:00:20000
13:97100117
8,005
2023-10.05
15.1010.11
15.53:00.12
15.53-10..12
15.53-1.122
17:00:1100
15.17.0.18
8:03
3547100.8
18:334000
00:77=0002
21.63:1:033
21.17:1:001
29.23:1:00
SVIN
8-0.
41.1710.001
26.07 0:113
21. 0.000
26.27:1:0.00
27 27.50:1:0.00
36.30-1-0.13
8=10
44.33-10001
30.77:00001
32.27
30.33110000
31,97.1:0000
29.47:12.86
KC=1
10,00::000
10:00:000
15.53.0.122
IC=2
31.00:00001
11.30:1:.33
0..27:20.29
15.67:10.331
15.00:1..322
14.00:1:024
4C=3
34.43:1:.40
8.93:1:0001
22.03:00.05
18.03:10.112
17.50:10.11
23.57:1::000
Table 12: Comparison with various FL algorithms in one round with 20 local epochs settings.
FeiLPA
FodNova
SCAFFOLD
FodAvg
FedProx
DENSE
24
115043007
15.35=0:15
13.40-0009
10.57-0000
1267-40.114
840.05
36.97
1883
19.13
1..14
19.70.10.17
33.13 1.047
2-0.1
28.1344:134
23.00-00:41
24.1044:.20
2403
6083-4033
42.50-00.12
0.00
40.47--0:222
2063-4-0.
40.67 1-004
FMNIST
840.5
0:.50:1::25
53.17 0:25
55.23 0.66
53.27:1-0.3
$4.20-10.52
6-.60-1-..5
2-1.0
75.47-00003
05.4740557
54.53-0.52
55.57
54.73-40.41
70.97400022
XC-2
29.67:0-.37
163340221
22.67-0.27
23.27220244
25.43-0.11
24.77200114
KG+3
34.37440799
24.93-00022
26.17 0.01
21.00--000
27.70--0003
20-43-0-022
8:005
23.37:1:001
11:33:10011
0.000
10.00:1:0000
10.77.10.00
1:06711:0022
4-01
25,07 +000
11:00+0005
12.33-0.10
12.67-t0.114
13
18-30---0006
84033
26.00-10.02
16.63-10.3
03.10:0.011
13.07-10.01
14,43
24.97 1.133
14:23:000
@=1.0
26.77+-0.10
17.00+00001
0.05
18.13+40004
182004-0001
26.07 1-0.1
IC=1
1-001
10.00::0.000
00000
10.20-1:.000
10.27
1-000-00.00
12:0--0000
10.25-0.00
11.10+0002
11.0040044
15.33-40014
82001
32.1310066
11:0:1006
0.30
93710000
101331000
12.10:10.19
850.0
68 45
24
07.77
37.70:1-.54
36.57:10775
62.00 1-0.4
71.77
8:033
90.3010001
68.17:10333
66 0.03
56.60110.115
66.00-10.21
35.37.10.11
MNIST
8=05
49.0300009
8150:10.14
76 0.113
70.57:10.22
47
86.07 007-00000
1.0
96.17440.01
86.60-1-.13
85.90 0,14
86.00
16.57-1-..15
1:000000
Kcl
10.0710011
10.13:1.0001
KG=2
53374-0611
17.4740.48
0.57
14.77400.14
13.47--0002
43.33- 11333+-0.110
KC+3
7227440.44
28.63 1.61
2.
31.40---2.01
30.57--2.50
44.30-1-0.22
13330013
000000000
85005
26.770011
13.7700044
0.
15.51410.12
15.57.2.112
8-0.1
28.0
17.07:10.05
0.000
19.3011:.01
19.70-1:.066
26.63 0.42
8:03
4500110.17
23:110004
2655
26.0010:000
27.57:00001
38.27-1-4774
SVHN
8-05
48000:21
02.20100.3
90.97:00144
43
8=10
62.23-10005
19.07
0.332
18.00-1:225
1..31-10.10
50.00 1.0.5
IC=1
23
9.3110033
07.07
16.601:0.11
15 15.50-1:0.12
7.70-100000
KC=2
27.7440.49
1133340.22
20.57 0.002
19.1744-.001
16.00-1-.27
18.53
25

---

Table 13: Comparison with various FL algorithms in one round with 50 local epochs settings.
Daxaset
Fattiion
FedLPA
FeaNova
SCAFFOLD
FedAvg
Feal
DENSE
8=001
19.352043
10. 10.13:0.00
15.87=0:16
18.53:0.35
12.97:0.15
10.70:0001
23.90:1::11
31.3344:220
8=0.1
40000:0001
30.40:::005
27.37=0023
25:33::::35
25:50:1:0.22
39.111100
8-0.1
62.5010441
43.6710011
42. 02.50=0.009
41.5020.00
42.23:20.11
57.000004
FMNIST
8:0,5
68:27:00000
55.91
05.27-00388
53.95-10.27
55.00:1:266
63.50:10.11
61.00:1:0009
60.77:00122
61,0011:15
66:00:0:066
13
00.50:1:00
01:03=0002
10.50:::000
11:771:007
4C=2
27.001:0.02
16.37:10000
23.00:0.:12
18.37:1:0.88
18.60:1:0.22
29.33:1:0.44
VC33
38.11:001
25
25 0.22
26.701:0:5
26.40:10.18
57.51
8-001
00:30:0000
12:27=0007
12.17::0000
8:005
1:.90
11.00:1:000
10,33
10.13:1:0000
11 11:20:1::000
7:61::001
8:01
19. 0.
15.17:00.09
03
12.00:::00
12.43:1:006
19.13:0:009
3-03
0.1
12.53440055
12.33=0002
11:93:4:011
12.90:001
CIFAR-10
8:05
27.57 5777-.22
1344::00000
12.30:00.00
12:47:10002
13.47:1:0002
25.40:1::233
8=1.0
00
15.47.0.11
15 15.30=0011
15:23-10115
15:53:00000
20 17:220000
KC-1
0150+00000
00.3000000
10,30 10:30=00000
10:30::0000
10.33-+:000
10:00:0000
10.03:00000
01.93:0.02
11:27:0001
85:07:1:008
00
12:00:::006
12.07=0002
12
11 90::0:00
21
D-001
14.1040.88
10.57 0.00
0.50=00000
9,33-40002
10.13.00011
1253-20.19
66.3310.33
19.001.33
8710.15
3-01
72.00+0.27
12
47.20=0.:2
30
0-03
87.00-00022
68. 08.30::0333
0.14
67. 07.10:20.09
66.17:0022
X2.47
MNIST
8:05
00
80 00:00:1000
18.1320000
79.37:00119
79 50:00.33
06.30-00.05
8-10
04
86.73.000
05143=0.11
860720.112
66.00:0116
23:0000
4C=1
01000:00011
10.17:10001
10.13:00001
10.13:1::01
10:00:1:001
9:801:0000
IC=2
71 0700000
23.53 1.00
02.93
22.311009
17:53:20122
42.97:10.17
KC-3
17:33
60 1.90
31 23.50
32.7742..27
23.40
17:30:011
0-001
8-006
22.071001
14.8740.112
0:
186740011
14.67400144
19:97:1054
(-0.1
45.3+11.0
22 02:40::211
22.97=0.12
22.47:20.01
24 24.30-::004
41 1147--4888
0+033
35.90:00.15
03.87:0.20
35.5040.135
34.43-10211
2950-8036
8-03
01.770001
10.93-0:13
38.334:0122
1011111
2-1.
40
56.00:0.115
54.03-00.28
55.33420.25
35.00:0.15
41 80488.911
4C=1
19.07+00.00
9.001:0.33
16.57=0.12
16.53:0.122
15:53:0.122
10:01:0000
36.03-20022
12.53.0.25
0.07
30.00:0:122
15:57:00.22
40.7742.211
9C-3
21007:0311
29.63=0.16
24.7340000
Table 14: Comparison with various FL algorithms in round with 100 local epochs settings.
DENSE
82001
17.0000
117310006
16. 10
19:0110:7
126710.12
10071000
8=005
36.772:0.11
18.0710335
22.67=0003
22.73:1:044
21 21.30:0:117
51.77::1114
9-011
35.0040.12
32.33-0.58
29.87:0.49
29.33-2:000
33.233 22
8:03
64.07:1:2288
47.77:10007
42.20:20001
43.33-10006
46.00:1:006
60.30:10017
FMNIST
8=0.5
68.73100110
57.01-00000
55.8720.38
56 10100.3311
58.6010043
64601000
8=10
76.07:1:.000
65.00:1:.05
61.67:0.35
65.13:10.14
65.00-10.111
85:00-10055
KC=1
10.57.0000
10:47:0.00
10.57::0000
13.2311:111
1000:1:000
20.01:0033
24.97:0.19
25.13::::25
21.50:1:029
26.30:11
IC=3
29.73-1021
26.07110:00
25.57
26.17:1:222
25.50:1:0.12
16.87-10.10
80001
10:30:00000
0.097
11.00:0000
21230001
3-005
1907
12.33:0.111
12.50-00:12
10.3::0000
12.6020.13
2-011
20.04:000
12.53:50.05
10.33
10.67::0000
11.87:0008
24.3035:000
3:03
28 30:0000000
11:63:00000
11 01.03
11:07:1:000
11 71:00:0001
26.23-1-336
CIFAR-10
000
12.0700000
02.13
11,00:0011
13.17:00111
28.00::0449
8=1.0
20.57:1000
14.00:1:009
030
13:97:10.100
15
29.37:1::.33
4C=1
01
10.2310000
10.23_0.00
10:23:::000
10:57:0000
10:00::000
KC-2
16 70:00
100000000
029
11:00::0011
11.97-20.0
13.67-20003
11:30:1:000
3-001
34 1010
13,012033
0.39
9,30::000
9,0000000
16.6110033
0-005
72.47=40007
32.30:20066
4.337
38.57:00335
20 40704:0449
55. 30-50-111 1.88
00.10
01-1000
76
3-03
053340004
68.77.000
0.14
67.1340011
67
MNIST
0-05
89 29.01:20.12
81.33
09.13
79.00:0023
79 79.50:2..3
88. 33.30330.31
8=10
13:0000
86.51.1000
85 05.87=0.00
8563-10088
86 17:1014
92:57-1:002
KC-1
11.27:4011
10300:0000
00:00
10.104001
10.13:0001
03.47=0.89
14.50.4..12
JC=3
29.43-2433
27552
12.4743311
20:24:.2
[-001
19:5043000
1590400.16
0.37
12.5740:22
11,004000
19.10:0:13
0-005
20-00-11.44
8-0.
36 366110227
22.37.1:23
14.1720.20
24.3:00077
25.03-10009
26.97:2025
0-0.3
5640000011
35.43-20.10
34.40=00015
35.17::0.10
34 34.00-00777
35.67-11555
SVHN
2-05
5563-14116
2000
40.33-0.05
57.47400011
57.07-00112
55.33-40622
9-1.0
65574001
5587700.27
55 0.266
$4.00:00.19
14.17:00334
62.50-10.11
xC11
16.77:0.22
10:3300000
13.8
15.6740.11
15.3--0.11
12.104
KC=2
41.87+-101
14 14.00:00.12
22,53-0:07
20777--0.06
13387-0026
41.43--11777
23.50:10044
30.20)-0.08
25:30:::000
Here, we present experiments similar to those in Table I but with different numbers of epochs (10, 20, 50, 100). The performance of our methods outperforms other approaches, as shown in Table I1 Table 12 Table 13, and Table 14 Without tuning the number of local epochs, our method consistently achieves high performance compared other baselines. In almost all the settings, our method outperform the state-of-the-arr baseline approach DENSE. We also note that DENSE
26

---

consumes more computing resources. shown in Table 5 Besides, it needs an extra data generation stage and an extra model distillation stage. Our method could get better results and consume fewer resources. What's s more, in Section 4.7 we also show that our method has the potential to extend to multiple-round settings, while it hard to extend the DENSE into multi-round settings.
G.3 Extreme setting, clients
Table 15: Comparison with various FL algorithms in one round when client number is 5.
Partition
FedI.PA
FedNova
SCAFFOLD
FedAvg
DENSE
3=0.01
48.13:10.28
26.03:0.077
30.77::0.49
30.80=0.34
1783::0:07
44.23:0.114
8=0.05
55.20=0017
23.40:0.16
30,80:0..77
29.90=0.12
20.43+0.16
46.17=00.09
8=0.1
59.27:0.12
33.47:0.16
37.7710445
32.57::0.98
58.73:20.15
FMNIST
8=0.3
73.13.10.00
53.1310..22
52.57.10.46
$2.03 0.55
49.90.1..333
63.40.10.06
8=0.5
74.17=00.02
60.27.00.33
60.13:1:0.7
59.97=1.14
61.67:40.35
72.03-+0.05
8=1.0
75.30.10.00
63.00.1.005
60.87.10.24
62.63
60.37.1.0.01
7..93
When the number of clients is set to 5, the experimental results for the FMNIST dataset are shown in Table 15 These results demonstrate that framework performs well even in extreme situations when the number of clients is relatively small.
G.4 Extreme setting, 3 = 0..01
Table 16: Comparison with various FL algorithms in one round with different epoch numbers and
0.001.
Daraset
epochs number
FAdLPA
FodNova
SCAFFOLD
FodAvg
FedProx
DENSE
10
14,57=0.04
10.60=00.01
10.53=0.001
10.60:20.01
13.10=00:01
10.00:
20
15.33:10.04
10.13:10.00
10.23:10.00
10.13:10000
12877:0.16
10.00+1..00
FMNIST
50
13.77.10.02
10.571.0.01
10.17110.00
10.5711..01
12.30.1.111
10.00.10.00
100
15.83-20.03
10.17:00.00
10.73:20.01
10.1740..00
13.2340021
10.00:1:0.00
200
14.53=0.00
10.07=00.00
10.10:10.00
10.07:+0.00
12.50-20112
10.00
10
11.50:10.00
10.27=00.00
10.17::0.00
10.27:10.00
10.33:40000
10.00:10.00
20
10.57=20.01
10.27:20.00
10. 10:00000000
10.27:+0.00
10.30=10.00
10.00:
CIFAR-10
10.77=20.01
10.23=10.00
10.33++0.00
10.23:+0.00
10.33::0.00
10.00
100
10.90:10.01
10.20:10.00
10.30:1:0000
10.23:1:0.00
10.57:10.01
10.00
10.27-10.00
10.23-10.000
10.27-10.00
10.37-10.01
10
14
10.07-10.01
12.17
11.83:10.05
12.17400112
9.90
20
19.53:20.33
10.07=20.01
12.07:20.07
13.37=20.08
12.37=0.12
9.27=20.00
MNIST
16.33
10.80:10.04
13.17-10.09
13.1
11.40:10.08
100
19.07 1..41
10.13.10.01
10.97 1.000
11.37.10.02
12.90.1..13
12.83.10.17
200
15.63-20.03
10.07=
11.13:00006
12.50:100004
11.83+0.11
9.27-+0.00
17.50:10.02
15.53:10.12
15.53:0.12
17.13:10:03
20
20. 101121
15.90.10.00
15.53
15.53.10.12
14.00-10.11
17.13-10.03
20.07-10.71
16.30-10.00
15.50:10.12
15.1310.116
14.03
15.1740.116
100
19.70=00.00
15,90=00.00
10:
15.53=+0.12
13.77
18:47400.05
200
19.13:2:.000
13.90:10.11
14.90-10.19
15.13:10.16
13.27440.06
15.23:40.16
Here, demonstrate that even when 3 0.001 and with different dataset and local epoch number settings, FedLPA has the potential to aggregate models effectively in extreme situations and produce superior results. These results are presented in Table 16
G.5 Experiments with FedOV and Co-Boosting
We compare with FedOv the state-of-the-art method which addresses label skews in one-shot federated learning. We the experiments with fair comparison (same model size) MNIST dataset with #C=2 partition setting. Table T7 shows that our method could be comparable with FedOV in some scenarios even when FedOv transmits the unknown label information the clients and utilizes the knowledge distillation. As the epoch number of local clients equals to 50, 50,100.200, FedLPA outperforms FedOV.
We also compare with Co-Boosting" the state-of-the-art distillation method. We run the experiments with fair comparison (same model size) the FMNIST dataset, and the rest of the settings the
27

---

Table 17: Comparison with FedOV on MNIST with #C=2.
epoch number
10
20
50
100
200
FedLPA
47.9
53.3710.61
71.07
71.07:0.335
69.6310.29
FedOV
71.0=0.25
70.27=0.39
69.23+0.31
65.83=10233
64.50:10.38
Table 18: Comparison with Co-Boosting on FMNIST.
B
0.01
0.05
0.1
0,3
0.5
PA
21.20=0067
54.27=0.38
55.33=0.06
68.20=0.04
73.33=0.06
Co-Boosting
10.21
18.97:1.44
1.81
83.37:00444
86.2110.31
same the default. The results in Table 18 show that when the 3 smaller than 0.1, our method outperforms Co-Boosting. Thus, with the increment of skewness, FedLPA shows significantly superior results.
In conclusion, our method could be comparable with FedOV and Co- Co-Boosting in some settings, even when they consume more computational resources shown in Appendix G.7
G.6 Communication overhead evaluation
5 shows the communication overhead evaluation a simple CNN with 5 layers on CIFAR-10 dataset. The results given based on the experiments. In this section, will give concrete example to show the details.
The communication bits are the number of bits that are transmitted between a server and client in directed communication. It reflects the communication efficiency of federated learning algorithms. Better algorithms should have lower communication bits. The default floating point precision is 32 bits in Pytorch.
A fully-connected neural network model example: We use a fully-connected neural network model with architecture 784-256-64-10 an example to show the calculation, which has 784 256 + 256 256 64 + 64 + 64 10 + 10 = 217930 floating point numbers, which is 6973760 bits or around 0.831 MB.
For a single directed communication from a client to the server or vice versa, the cost for FedAvg. FedProx, FedNova, and DENSE is 0.831 MB each. SCAFFOLD costs 1.662 MB for the same communication, which is double the amount of the others.
For single communication from client to the server, our method requires additional upload of A and Bx. which contain 785 785 + 256 256 + 257 257 + 64 64 65 65 + 10 10 756231 floating point numbers in total. Note, A& and B3 are symmetric matrices, triangular part of them, reducing the total to roughly 756231/2 = 15.5 floating point numbers as about 1.442 MB. Therefore, our approach costs 2.272 MB for the one directed communication, which is 2.734 times as FedAvg, FedProx, and DENSE, and 1.367 times as SCAFFOLD.
only need upload the upper
A CNN model example: We use another example using CNN to show the communication overhead. For example, we have one model, the first layer is nn.Conv2d(1, channels, 6 output channels, and a 5x5 kernel size; the second layer is nn.Conv2d(6, 8, 5), means there are input channels, 8 output channels, and 5x5 kernel size.
5), means there are input
The parameter count for the first layer is 1x6x5x566=156 Note that A and B are both symmetric matrices. Thus, the additional parameters for A and B for each kernel would be 5x5, and estimating the covariance for biases without decomposition results in a size of 6x6. Therefore, the additional parameters for this layer are 201 (A,, 6x((5x5-5)/2+5) B,, (5) 6)/++6.
The parameter count for the second layer is 8x6x5x5+8-1208. Therefore, the additional parameters for this layer 1476
These two examples all follow the theory: the communicated parameters A, B and M are approxi- mately 2x of the number of all parameters in the model 0.
28

---

Table 19: Running time and computation overhead evaluation. FeuNoowsCCAFFOLDffedlly
FedLPA
FedProx
DENSE
FedOV
Co Co-Boosting
65mins
50mins
75mins
400mins
150mins
700mins
Table 20: Experiments with ResNet-18.
8
Fed
Nova
SCAFFOLDD
FedAvg
FedProx
Dense
0.1
23.62::0.51
12.16:10.23
10.0740.004
13.8710.26
12.04:0.16
21.45:10.60
0.3
27.43:10.04
01.75
10.08
10.01 10007
12.97:0.17
27.10
0.5
10.10 14
13.81
12.75+0.11
10.61
11.45+0.23
29.04++0.30
The communication overhead will increase linearly using FedLPA when we change the client numbers to 20 and 50.
However, as Figure demonstrates, to achieve the same performance as FedLPA, FedAvg, FedNova, SCAFFOLD, and FedProx require more communication rounds, resulting in heavier data transfer burden on the system.
G.7 Running time and computation overhead evaluation
The running times of different algorithms, using simple CNN on the CIFAR-1O dataset, are summarized in Table In this experiment, there are 10 clients, each running 200 local epochs with only one communication round. Our device is single 2080Ti GPU. Compared to the state-of-the-art methods FedOV, DENSE and Co-Boosting, our method is efficient and slightly slower than the fastest algorithm. Notably, DENSE consumes almost times the computational resources, as the knowledge distillation method is computationally intensive and resource-demanpe even uses more time. It's important to note that while our method is efficient, it also yields almost always the best results. In our paper, we mainly adopt the most-cited non-IlI FL benchmark (https://github.oo https://ithub..com//taaaaaCC
Co-Boosting
-Bnnch to get fair comparison of FedLPA and other baselines. The reason why the computation cost of FedProx is higher than FedAvg may be that the FedProx adds a l2 regular term to make local updates around the global mode, which adds more computing overhead. Besides, using the original codebase from FedPorx also consumes more time than FedAvg and FedNova, under the above non-III FL. benchmark.
Our methods guarantee that the computation result overhead will increase almost linearly using FedLPA when we change the client numbers to 20 and 50.
G.8 Experiments with more complex neural network structure
We do the experiments with FedL_P the experiment setting in the paper using the more complex network, ResNet-18 18 (95), with five random seeds on the CIFAR-10 10 dataset. We set the parameters with 8=0.1. 1. 0.3 and 0.5 with 10 clients. The results are shown in Table 20] From the results, we could see that using ResNet-18, our method still gets better performance compared to other baselines.
We do the experiments with FedLPA on the same experiment setting in the paper using the more complex network, VGG-9 (96). with five random seeds the FMNIST dataset. We set the parameters with 8=0.1, 1, 0.3 and 0.5 with 10 clients. The results are shown in Table From the results, we could that using VGG-9, our method still performs better than other baselines.
Based on the results of R8 and VGG-9, Our method has the potential to be applied to more complex models.
G.9 Experiments with more complex datasets
We do the experiments with FedLPA on the same experiment setting the paper using ResNet-18 18 with five random seeds on the CIFAR-100 65 dataset. The results are shown in T2ble We can see that even with the complicated dataset CIFAR-100, our method could also get satisfactory results in the federated one-shot setting. Besides, we also have added the experiments on EMNST 997 using
29

---

Table 21: Experiments with VGG-9.
PedLPA
FedNova
SCAFFOLD
FedAvg
FedProx
Dense
58.4841.33
28.77+2.03
32.45+0.12
33.71+0.16
31.78+00.0
01.76+00.28
0.3
71.98+1.72
53.78+00.2
55.00+1.07
54.61 +0.02
52.79+00.00
70.10+1.445
0.5
79.02.10.81
62.31
61.75.0..4
63.1811.71
62.55:1.117
76.20=11110
Table Experiments with CIFAR-100 using Fedl.PA PA..
FedAvg
FedLPA
0.1
1.31
15.11
0.3
1.75+0.10
18.8240.71
0.5
1.38#0.11
21.77+:0.03
simple-CNy CNN with 10 clients and five random seeds. We do the experiments EMNIST-mnist and EMNIST-letters. The results are shown in Table
In addition to these, we conduct experiments with Inn--emageeNet 998 with ResNet- 18 with 10 clients and five random seeds. The results are shown in Table 24
G.10 Ablation experiments analyzing the number of approximation iterations of FedLPA
The proposed method is composed of multiple approximations: 1) empirical Fisher to approximate the Hessian. 2) boock-daagonal Fisher matrix instead of full, 3) approximating global model parameter M with optimization problem in Eq. 14
Empirical Fisher approximate the Hessian:
Although empirical Fisher has been successfully applied in many methods and yielded good results, discussions concerning the approximation error of empirical Fisher are limited. Fortunately, previous work [99] provides detailed critical discussion of the empirical Fisher approximation.
i. Fisher to approximate the Hessian:
When the loss function represents exponential family distribution, the Fisher is well-justified approximation of the Hessian, and its approximation error can be bounded in terms of residuals. The accuracy of this approximation improves the residuals diminish and is exact when the data is perfectly fitted.
ii Empirical Fisher to Fisher:
It's noted that the Fisher and empirical Fisher coincide near minima of the loss function under two conditions:
A. The model distribution closely approximates the data distribution.
B. sufficiently large number of samples allows both the Fisher and empirical Fisher converge to their respective average values in the population.
In practical environments, especially condition 1, might not hold, causing bias between empirical Fisher and Fisher. However, empirical Fisher still contains effective covariance information. In second-order optimization methods, the covariance information in empirical Fisher can adapt to
Table 23: Experimental with EMNIST.
Dataset
Partitions
FedLPA
FedAvg
3=0.1
74.23+3.10
57.63+22.30
EMNIT-mnist (10 classes)
6=0.3
86.55+0.24
6..32.11.77
3=0.5
91.75+10.26
82.71-10.96
B=0.1
26.34.0.71
16,22:0.38
EMNIT-letters (37 classes)
B=0.3
31.75+0.03
25.5110.44
B=0.5
33.78+00.14
26.34+0.07
30

---

Table 24: Experiments with Iman-lmangeNett using ResNet-18
B
FedLPA
Dense
FedAvg
0.1
17.02=1.40
1..8
3.72.
0.3
27.80=2.10
24.91 11.6
8,41 10087
0.5
1..1
29.43
12.07 #1.92
Table 25: Experiments for the approximation study.
Number of iterations
Occuracy(5=0.1)
Accuracy(_5=0.)
Accuracy(3=0.5)
Computation(s)
1000
52.81+0.71
60.31-0.23
72.11+0..57
3
5000
59.70++0.32
68.09-0.30
74.27+0.11
15
10000
55.33:+0.06
68. 68.20==0.00
73.33++0.06
30
20000
58.41::0.05
68.11=00.07
73.51
60
the gradient noise in stochastic optimization. Nevertheless, referencing the work [54], we can use the model's predictive distribution obtain unbiased estimate of the true Fisher at the same computational cost as empirical Fisher.
(2) Block-diagonal Fisher matrix to approximate the full one: The work [100] provides detailed evaluation and testing of using block-diagonal Fisher to approximate the full one. Firstly, Chapter 6.3.1 "Interpretations of this approximation' in the paper (100) indicates that using block-wise Kronecker-facee factored Fisher closely approximates the full Fisher. Although there is bias term, this term approximates zero when there sufficient samples. Furthermore, the paper examines the approximation quality of block-diagonal Fisher compared with the true Fisher and suggests that block-diagonal Fisher captures the main correlations, while the remaining correlations have minimal impact on the experimental results.
(3) Besides, we have added some experiments for more ablation studies with our method on the same experiment setting the paper with five random seeds with clients. conducted experiments on FMNIST datasets with 3=0.1, 0.3 and 0.5. The results are shown in Table 25 We show the experiments analyzing the number of approximation iterations. With the experiment results, we could know that 5000 iterations are enough to get the ideal results. By default, we use 10000 iterations.
We also show that the computation time for the approximation linear with the number of approxi- mation iterations in the last column of Table 25
Additionally, it's worth noting that concerning Laplace approximation, the analysis 101 that the error of Laplace approximation is inversely proportional to the input dimension a with be inferred that in our method, for each layer of the
suggests
O(n According to this conclusion, neural network, the error of Laplace approximation inversely proportional to its width. When the neural network is infinitely wide, the approximation error tends towards zero.
G.11 Artifact details
We have uploaded the codebase containing all the methods compared in our paper. Setting up the environment is relatively straightforward with the provided readme file. If you refer to the scripts folder, you will find all the bash scripts necessary reproduce the tables and figures from our experiments.
The experiments.sh script covers the experiments in Table Table I1 Table 12 Table 13 and Table [4, Running these experiments on single 2080Ti GPU will take approximately days. Specifically, Table| will take about 35 days.
The experiments client.sh script covers the experiments in Table requiring approximately 40 days on single 2080Ti GPU.
The sxperiments_coorskt script covers the experiments in Table which can be completed in 2 days.
The experiments dh script covers the experiments in Table requiring approximately day on single 2080Ti GPU.
31

---

The experiments fedavg with attack.py and experiments fedlpa with attack.py covers the experi- ments Table requuiri approximately I day on single 2080Ti GPU.
The experiments eettrnttt__ttttee____eeeeee extrreme_clientsssl mately 4 days of GPU processing.
extreme clients.sh script covers the experiments Table 15 and requires approxi-
The setrriments_xxtremmeen script reproduces the experiments in Table 16 and takes about 10 days.
The experiments of Table 17 and Table 18 take about 8 days.
The experiments_emninee script covers the experiments in Table 23 and takes about day.
Running experiments eecerripenttsmulpplle.... multiple_round.sh round.sh will yield the results as shown Figure and this process takes about day.
The experiments for Table 6 and Table 7 will take about day. The experiments for Table 25 and Table will also take about day.
To generate the t-SNE visualizations shown in Figure experiments.py script with the "alg=tsne" option.
Figure 3 and Figure 4 you can use the
In total, reproducing all the experiment results in this paper will require about 185 days for GPU processing.
32

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
Justification:
The paper discusses the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
33

---

Justification:
The paper provides the full set of assumptions and complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: The paper fully discloses all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
34

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes].
The paper provides open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification: The paper specifies all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes].
Justification: The results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: For each experiment, the paper provides sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes].
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: There is societal impact of the work performed.
Guidelines:
36

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA].
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
37

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA].
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
38

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "rMcJRxXCHJ", "reviewer": "Reviewer_F77H", "summary": "This paper proposes a one-shot Federated Learning (FL) method, denoted as FedLPA, to address heterogeneous data distribution among clients. FedLPA does not demand auxiliary datasets or private label information during aggregation on the server side. To achieve this, FedLPA infers the posteriors by leveraging the Fisher information matrix of each layer in local models using layer-wise Laplace approximation and aggregates these to train the global model. Abundant experiment results demonstrate the efficacy of FedLPA compared to conventional FL methods under the one-shot setting.", "questions": "As I mentioned in the weakness, the authors should compare the proposed method with DP FL and prototype-based FL rather than conventional FL methods. I believe the experiments in the paper do not provide a fair comparison. \n\nAccording to the authors in Supplementary Sec.L, the security level of the proposed method is similar to that of FedAVG, meaning it is also vulnerable to attacks targeting FedAVG. Does the only advantage of the proposed method is that attackers have fewer opportunities to strike since they only communicate with the server once?", "limitations": "The topic discussed within the paper is highly related to privacy protection and the authors show a novel method to deal with it.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of the paper is easy to follow.\n2. Instead of measuring co-relations between different layers, the proposed method only approximates layer-wise Fisher to get a good trade off.\n3. Extensive experiment results show the superiority of FedLPA under the one-shot FL setting.", "weaknesses": "To be honest, I do not buy the one-shot Federated Learning (FL) setting. This setting goes against the idea of FL. But this does not affect my rating.\n1. There are some typo problems.\n2. Authors should compare the proposed method with differential privacy (DP) FL or prototype-based methods rather than conventional FL methods. All these approaches address communication security but from different perspectives.\n3. Would the proposed method also show superior performance on more challenging datasets, like tinyimagnet and office-home?\n4. Any proof to show that \"computing the co-relations between different layers brings slight improvement\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer F77H, thanks for your comments, which helped us improve our paper. The answers to all your questions are as follows:\n\n>  **Q1**: There are some typo problems.\n\n**Answer**: Thanks for pointing that out and thanks for your efforts in reviewing our paper. We found the following typos and will revise if we find more.\n\n\nSec3.7 We->we\n\nSec4.1 benchmark->benchmarks\n\nAppendix F in one-shot -> in the one-shot\n\nAppendix M.13 proportion -> proportions\n\n>  **Q2**: Authors should compare the proposed method with differential privacy (DP) FL or prototype-based methods rather than conventional FL methods. All these approaches address communication security but from different perspectives.\n\n>And\n\n> As I mentioned in the weakness, the authors should compare the proposed method with DP FL and prototype-based FL rather than conventional FL methods. I believe the experiments in the paper do not provide a fair comparison.\n\n**Answer**: Beyond the privacy protection and the reduction on the attack surface, the one-shot FL framework also tries to deal with the substantial communication overhead and higher demand for fault tolerance throughout the rounds. In Appendix L.1, we show the experiment of DP-FedLPA and DP-FedAvg in the one-shot setting. Specifically, since the sensitivity of the data sample distribution after the normalization is 1, we add Laplacian noises with $\\lambda=\\frac{1}{\\epsilon}$. We set $\\epsilon=\\{3,5,8\\}$ that provides modest privacy guarantees since normally  $\\epsilon \\in (1,10)$ is viewed as a suitable choice. \n\nBesides, we have added the experiments using the same experiment setting in the paper on the FMNIST dataset to compare DP-FedAvg (multiple-round) as Appendix L.1 with our FedLPA (one-shot), show the round results of how many rounds DP-FedAvg needs to achieve the same test performance. \n\n| $\\epsilon$\\Partitions | 0.1 | 0.3 | 0.5 |\n|------------|-----|-----|-----|\n| 8        | 11  | 10 | 8  |\n| 5        | 11  | 9  | 8  |\n| 3        | 12  | 9  | 7  |\n\nThe results show that DP-FedAvg needs about 10 rounds of communication to achieve the same test performance, compared to our one-round FedLPA. Combined with our previous results in Table 4 and Table 7, our FedLPA could save the communication and computation overhead and combine with the DP method to mitigate the potential privacy leakage. Based on the above settings, DP-FedAvg needs at least **3x communication overhead** and **5x computation overhead**. While DP-FedAvg needs multiple rounds to get similar accuracy, DP-FedAvg maybe vulnerable to more privacy attack methods due to the multiple queries, such as curvature-based privacy attacks.\n\n\n>  **Q3**: Would the proposed method also show superior performance on more challenging datasets, like tinyimagnet and office-home?\n\n**Answer**: In our paper, our FedLPA shows good performances on MNIST, FMNIST, CIFAR-10, CIFAR-100, SVHN, EMNIST datasets. We further add the experiments on the same experiment setting in the paper using ResNet-18 on Tiny-ImageNet. The results show that our method has the potential to deal with large datasets.\n\n| Partitions | FedLPA        | Dense         | FedAvg         |\n|------------|------------|-------------|--------------|\n| 0.1        | 17.02$\\pm$1.40 | 15.88$\\pm$1.96 | 3.72$\\pm$1.44  |\n| 0.3        | 27.80$\\pm$2.10 | 24.91$\\pm$1.65 | 8.41$\\pm$0.87  |\n| 0.5        | 30.14$\\pm$1.25 | 29.43$\\pm$0.72 | 12.07$\\pm$1.92 |\n\nDue to the rebuttal time limit, we leave the implementation on office-home dataset as future work. However, we believe that the FedLPA performs well on office-home dataset,  since it shows the potential to deal with challenging data on Tiny-ImageNet dataset.\n\n>  **Q4**: Any proof to show that \"computing the co-relations between different layers brings slight improvement\"?\n\n**Answer**:  Paper [1] we cited in our paper provides a detailed evaluation and testing of using block-diagonal Fisher to approximate the full one. Firstly, Chapter 6.3.1, \"Interpretations of this approximation\" in the paper [2] indicates that using a block-wise Kronecker-factored Fisher closely approximates the full Fisher. Although there is a bias term (due to the approximation in our Appendix Equation 30), this term approximates zero when there are sufficient samples. Furthermore, the paper examines the approximation quality of block-diagonal Fisher compared with the true Fisher and suggests that block-diagonal Fisher captures the main correlations, while the remaining correlations have a minimal impact on the experimental results.\n\n\nWe will add the above analysis in the appendix on the camera-ready version.\n\n[1] Ritter, Hippolyt, Aleksandar Botev, and David Barber. \"A scalable laplace approximation for neural networks.\" 6th international conference on learning representations.\n\n[2] Martens, James. Second-order optimization for neural networks. University of Toronto, 2016.\n\n>  **Q5**:  According to the authors in Supplementary Sec.L, the security level of the proposed method is similar to that of FedAVG, meaning it is also vulnerable to attacks targeting FedAVG. Does the only advantage of the proposed method is that attackers have fewer opportunities to strike since they only communicate with the server once?\n\n**Answer**:  In fact, our approach can reduce the attack surface as attackers have fewer opportunities since they only communicate with the server once. Beyond that, in the one-shot setting, the FedLPA could also deal with the substantial communication overhead and higher demand for fault tolerance throughout the rounds.\n\nFedAvg is vulnerable to privacy attacks with huge communication overhead. However, FedLPA is compatible with existing preserving approaches (i.e., DP) to achieve an even higher privacy level and strike a balance between computation, communication overhead, and performance. Note that our paper mainly focuses on the efficiency perspective to improve the performance of one-shot FL with inferior consideration of the security perspective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FCob1rE0J1", "reviewer": "Reviewer_XECp", "summary": "The paper \"FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation\" introduces FedLPA, a novel one-shot federated learning method that addresses challenges associated with high statistical heterogeneity in non-identical data distributions. The framework uses layer-wise posterior aggregation based on the empirical Fisher information matrix, allowing for the accurate capture and aggregation of local model statistics into a global model. The paper claims that FedLPA improves learning performance significantly over state-of-the-art methods across several datasets without requiring auxiliary datasets or exposing private label information.", "questions": "I have no question.", "limitations": "The authors have acknowledged some limitations of their work, but further discussion would enhance the robustness of the paper:\n\n1. **Complexity and Practicality**: The complexity of the FedLPA framework may hinder practical implementation. Providing more detailed guidelines or potential simplifications could make the approach more accessible for real-world applications. Additionally, discussing potential trade-offs between complexity and performance would be beneficial.\n2. **Scalability**: The paper does not sufficiently address the scalability of the proposed methods in environments with many clients or large graphs. Detailed evaluations of the computational and communication overheads involved in scaling the methods would help understand their practical feasibility.\n3. **Privacy Risks**: A deeper analysis of potential privacy risks and mitigation strategies is necessary, particularly in federated learning settings where data privacy is a major concern. Discussing how privacy-preserving techniques can be integrated into the proposed framework would strengthen the paper.\n4. **Experimental Validation**: While the experiments are comprehensive, further validation on larger-scale datasets and more diverse GNN architectures would provide stronger empirical support for the proposed methods. Expanding the experimental scope would help demonstrate the applicability of the methods in various real-world scenarios.\n\nOverall, the paper makes valuable contributions to federated learning for graph data, but addressing the mentioned weaknesses would further enhance its robustness and applicability.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Originality**: The introduction of layer-wise posterior aggregation using the empirical Fisher information matrix is a novel approach in the context of one-shot federated learning. This method effectively addresses the challenge of non-IID data distributions.\n2. **Quality**: The paper provides a rigorous theoretical foundation for the proposed method, including convergence proofs and detailed mathematical formulations. The extensive experimental results on various datasets further support the claimed improvements in learning performance.\n3. **Clarity**: The paper is well-structured and clearly written, with thorough explanations of the methodologies and theoretical concepts. The inclusion of figures and tables helps in understanding the experimental results.\n4. **Significance**: By improving the performance of one-shot federated learning under non-IID conditions, the proposed method has significant implications for practical applications where data privacy and communication efficiency are critical concerns.", "weaknesses": "1. **Implementation Complexity**: The use of layer-wise posterior aggregation and the empirical Fisher information matrix introduces significant complexity. The practicality of implementing FedLPA in real-world settings could be challenging without detailed guidelines or simplifications.\n2. **Scalability**: The paper does not adequately address the scalability of the proposed method to large datasets or a high number of clients. Evaluating the computational and communication overheads in such scenarios is necessary to understand the practical feasibility of FedLPA.\n3. **Privacy Considerations**: While the method claims to preserve data privacy, the paper lacks a detailed discussion on potential privacy risks and mitigation strategies, which is crucial for federated learning applications.\n4. **Experimental Scope**: The experiments, although comprehensive, are limited to a few datasets and simple neural network structures. Additional validation on larger and more complex datasets, as well as a variety of neural network architectures, would provide stronger empirical support for the method's generalizability.\n5. **Parameter Sensitivity**: The performance of FedLPA may be sensitive to the choice of parameters, such as the noise levels for the empirical Fisher information matrix. A detailed analysis of parameter sensitivity and guidelines for selecting appropriate parameter values would enhance the robustness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer XECp, thanks for your comments, which helped us improve our paper. The answers to all your questions are as follows:\n\n>  **Q1**:  **Implementation Complexity**: The use of layer-wise posterior aggregation and the empirical Fisher information matrix introduces significant complexity. The practicality of implementing FedLPA in real-world settings could be challenging without detailed guidelines or simplifications.\n\n>And\n\n> **Complexity and Practicality**: The complexity of the FedLPA framework may hinder practical implementation. Providing more detailed guidelines or potential simplifications could make the approach more accessible for real-world applications. Additionally, discussing potential trade-offs between complexity and performance would be beneficial.\n\n\n**Answer**：The theoretical analysis of the layer-wise posterior aggregation may be complex, however, the practical implementation is not complicated. Besides, we have provided several functions with APIs to ensure non-experts can adopt our FedFPA framework within a few lines of code. In detail, in our submitted source code zip file, we provide the APIs for the users to use our method in line 541 of experients_our.py. We also include the artifact details in our paper and provide all the script files to reproduce the results of our paper.\n\nIn our paper, we also show that our FedLPA strikes a balance between the computation overhead and the performances and incurs solely 30% more computation time compared to the simple FedAvg method while increasing the test accuracy by **up to 35%** in some settings, as shown in Table 1 and Table 4.\n\n>  **Q2**:  **Scalability**: The paper does not adequately address the scalability of the proposed method to large datasets or a high number of clients. Evaluating the computational and communication overheads in such scenarios is necessary to understand the practical feasibility of FedLPA. \n\n \n**Answer**：In our paper, our FedLPA shows good performances on MNIST, FMNIST, CIFAR-10, CIFAR-100, SVHN, EMNIST datasets. We further add the experiments on the same experiment setting in the paper using ResNet-18 on Tiny-ImageNet. The results show that our method has the potential to deal with large datasets.\n\n| Partitions | FedLPA        | Dense         | FedAvg         |\n|------------|---------------|---------------|----------------|\n| 0.1        | 17.02$\\pm$1.40 | 15.88$\\pm$1.96 | 3.72$\\pm$1.44  |\n| 0.3        | 27.80$\\pm$2.10 | 24.91$\\pm$1.65 | 8.41$\\pm$0.87  |\n| 0.5        | 30.14$\\pm$1.25 | 29.43$\\pm$0.72 | 12.07$\\pm$1.92 |\n\nWe also add the experiments with more clients in the same experiment setting of the paper using FMNIST datasets. The results show that our method could support up to 200 clients.\n\n| Partitions\\Clinet number | 10             | 20             | 50             | 100           | 200           |\n|--------------------|--------------|--------------|--------------|--------------|-------------|\n| 0.1                    | 55.33$\\pm$0.06 | 57.37$\\pm$0.05 | 57.03$\\pm$0.00 | 54.80$\\pm$0.13 | 54.17$\\pm$0.26 |\n| 0.3                    | 68.20$\\pm$0.04 | 71.30$\\pm$0.03 | 66.70$\\pm$0.23 | 66.28$\\pm$0.45 | 64.52$\\pm$0.08 |\n| 0.5                    | 73.33$\\pm$0.06 | 74.07$\\pm$0.00 | 71.13$\\pm$0.00 | 70.72$\\pm$0.09 | 70.05$\\pm$0.27 |\n\n\nThe computational and communication overheads are shown in Table 4 in our paper. As the number of clients increases, the computational and communication overheads increase linearly.\n\n>  **Q3**:  **Privacy Considerations**: While the method claims to preserve data privacy, the paper lacks a detailed discussion on potential privacy risks and mitigation strategies, which is crucial for federated learning applications.\n\n**Answer**：In Appendix L, we give a detailed discussion about privacy concerns for our FedLPA. We show that our method has the same privacy level as the baselines, counteractting the iDLG attack. Our method could also be compatible with existing privacy-preserving technologies (i.e., DP). At the end of Appendix L, we also provide a concrete example of privacy attacks to show how FedLPA preserves data privacy.\n\n>  **Q4**:  **Experimental Scope**: The experiments, although comprehensive, are limited to a few datasets and simple neural network structures. Additional validation on larger and more complex datasets, as well as a variety of neural network architectures, would provide stronger empirical support for the method's generalizability.\n\n>And\n\n> **Experimental Validation**: While the experiments are comprehensive, further validation on larger-scale datasets and more diverse GNN architectures would provide stronger empirical support for the proposed methods. Expanding the experimental scope would help demonstrate the applicability of the methods in various real-world scenarios.\n\n**Answer**：In our paper, our FedLPA shows good performances on MNIST, FMNIST, CIFAR-10, CIFAR-100, SVHN, EMNIST datasets. We further did the experiments with  Tiny-ImageNet, as shown above. \n\nIn Appendix M.10, we also show the results using more complex network structures such as ResNet-18. \n\nIn this paper, we conduct the experiments on CV tasks without focusing on the graph neural network.\n\n>  **Q5**: **Parameter Sensitivity**: The performance of FedLPA may be sensitive to the choice of parameters, such as the noise levels for the empirical Fisher information matrix. A detailed analysis of parameter sensitivity and guidelines for selecting appropriate parameter values would enhance the robustness of the method.\n\n**Answer**: In our paper, we do not have the noise level settings for the empirical Fisher information matrix. We only have one hyper-parameter, $\\lambda$ from Eq. 33, which controls variances of a priori normal distribution and guarantees $A_k$ and $B_k$ are positive semi-definite. \n\nAll other Laplace Approximations are sensitive to the hyper-parameter  $\\lambda$ based on their experimental results, **but Table 3 shows that our approach is relatively robust**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejdhIcYfCL", "reviewer": "Reviewer_4qUr", "summary": "This paper proposes FedLPA, a novel one-shot federated learning method that uses layer-wise posterior aggregation. It aggregates local models to obtain a more accurate global model without requiring extra datasets or exposing private label information. The key innovation is using layer-wise Laplace approximation to efficiently infer the posteriors of each layer in local models, then aggregating these layer-wise posteriors to train the global model parameters. Extensive experiments show FedLPA significantly improves performance over state-of-the-art methods across several metrics, especially for non-IID data distributions.", "questions": "please see the weakness", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tIt achieves good performance with only a single round of communication between clients and the server, reducing communication overhead and privacy risks. It performs well on heterogeneous data distributions across clients.\n2.\tFedLPA doesn't need additional public datasets.\n3.\tThe paper provides a convergence analysis showing a linear convergence rate for the global model optimization.", "weaknesses": "1.\tThe method relies on multiple layers of approximation - empirical Fisher to approximate the Hessian, block-diagonal Fisher instead of full, and approximating global model parameters through optimization. Each approximation introduces some error. These compounding approximations could potentially lead to suboptimal global models. However, the paper's empirical results suggest that in many practical scenarios, these approximations still lead to good performance. Nonetheless, a more thorough theoretical analysis of these approximation errors and their impact would strengthen the paper.\n2.\tAlthough more efficient than some baselines, FedLPA still requires more computation than simpler methods like FedAvg.\n3.\tStoring and transmitting the block-diagonal Fisher matrices for each layer increases memory usage and communication costs compared to methods that only share model weights. For very large models or with many clients, the increased communication overhead from sharing Fisher matrices could become significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4qUr, thanks for your comments, which helped us improve our paper. The answers to all your questions are as follows:\n\n>  **Q1**: The method relies on multiple layers of approximation - empirical Fisher to approximate the Hessian, block-diagonal Fisher instead of full, and approximating global model parameters through optimization. Each approximation introduces some error. These compounding approximations could potentially lead to suboptimal global models. However, the paper's empirical results suggest that in many practical scenarios, these approximations still lead to good performance. Nonetheless, a more thorough theoretical analysis of these approximation errors and their impact would strengthen the paper.\n\n**Answer**: In our paper, note that we have three approximations listed as follows:\n\n(1) empirical Fisher to approximate the Hessian\n\nIn our Appendix E, we show the theoretical analysis of this approximation error.\n\n(2) block-diagonal Fisher instead of full Fisher\n\nPaper [1], which we cited in our paper, provides a detailed evaluation and testing of using block-diagonal Fisher to approximate the full one. Firstly, Chapter 6.3.1, \"Interpretations of this approximation\" in the paper [2] indicates that using a block-wise Kronecker-factored Fisher closely approximates the full Fisher. Although there is a bias term (due to the approximation in our Appendix Equation 30), this term approximates zero when there are sufficient samples. Furthermore, the paper examines the approximation quality of block-diagonal Fisher compared with the true Fisher and suggests that block-diagonal Fisher captures the main correlations, while the remaining correlations have a minimal impact on the experimental results.\n\n(3) approximating global model parameters through optimization\n\nIn our Appendix J, we show the convergence analysis of our method.\n\nIn summary, the approximations have a negligible effect on the final test accuracy. Some experiment results are shown in Table 22 of Appendix M.12. \n\nWe will summarize these in the appendix once accepted.\n\n[1] Ritter, Hippolyt, Aleksandar Botev, and David Barber. \"A scalable laplace approximation for neural networks.\" 6th international conference on learning representations, ICLR 2018-conference track proceedings.\n\n[2] Martens, James. Second-order optimization for neural networks. University of Toronto (Canada), 2016.\n\n\n>  **Q2**: Although more efficient than some baselines, FedLPA still requires more computation than simpler methods like FedAvg.\n\n**Answer**: Our method FedLPA only introduces 30% more computation time compared to the simple FedAvg while increasing the test accuracy by **up to 35%** in some settings, as shown in Table 1 and Table 4. As shown in Table 17, the state-of-the-art Dense will use **6.15x times the computation time of our FedLPA**, while our method performs better in most cases considering test accuracy. Co-Boosting, another distillation method, will use **10.77x times the computation time of our FedLPA**. Our method is also faster than FedProx and FedOV. Our method is compatible with FedOV and Co-Boosting and performs much better than FedProx.\n\nOur FedLPA strikes a balance between the computation overhead and the performances.\n\n\n>  **Q3**: Storing and transmitting the block-diagonal Fisher matrices for each layer increases memory usage and communication costs compared to methods that only share model weights. For very large models or with many clients, the increased communication overhead from sharing Fisher matrices could become significant.\n\n**Answer**:  Our FedLPA strikes a balance between the communication overhead and the performances. As shown in Table 4, FedLPA also has a communication overhead of only 2x times that of FedAvg. The communication overhead is similar between our FedLPA and the popular SCAFFOLD. In some settings, FedLPA could increase the performance up to 35% compared to FedAvg and SCAFFOLD.\n\nIn our paper, we also give detailed examples of the communication overhead of our FedLPA in Appendix M.8.\n\nFor all the datasets, as the number of clients increases, the communication overhead will also increase linearly. Intuitively, for very large models or with many clients, our communication overhead is solely about two times of FedAvg. Compared with the computation overhead in the one-shot setting, this is acceptable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a new one-shot federated learning method. The key is a novel one-shot aggregation method with layer-wise posterior aggregation called FedLPA. It addresses the problem of heterogeneous data distribution among clients. FedLPA infers the posteriors by leveraging the Fisher information matrix of each layer in local models using layer-wise Laplace approximation and aggregates these to train the global model.

Reviewers recognize the importance of the problem, contrition/novelty of the work and the empirical study. Most of the concerns raised by the reviewers were addressed by the authors, and two reviewers raised their scores. The final scores are all favorable albeit with a varying degree of support (5, 6, 6).

4qUr (rating 5) has some reservations regarding the risk of privacy leakage. The same concern is shared by F77H. In the final discussion period XECp states that 

> Privacy issue is an inherent drawback of federated learning methods. The main contribution is sufficient for this paper. To address the privacy problem, there are numerous techniques focused on privacy protection.

The AC believes that the contribution and novelty of the work is interesting enough to be shared with the community. For the final version please include the points clarified in the rebuttal and add a discussion regarding possible privacy leakage.
Decision: Accept (poster)