Harnessing Heterogeneous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Authors: Mahendra Singh Thapa
Rui Li
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Bayesian Inference
Variational Inference
Primary area: deep_learning

Abstract: Personalized federated learning (PFL) based on Bayesian approach tackle the challenges from statistical heterogeneity of client data by computing a personalized posterior distribution over the parameters of each client's local model and constructing a global distribution by aggregating the parameters of these personalized posteriors. However, the heuristic aggregation methods introduce strong biases and result in global models with poor generalization. We thus propose a novel hierarchical Bayesian inference framework for PFL by specifying a conjugate hyper-prior over the parameters of the personalized posteriors. This allows us to jointly compute a global posterior distribution for aggregation and the personalized ones at local level. This hierarchical Bayesian inference framework achieves elegant balance between local personalization and global model robustness. Extensive empirical study shows that by effectively sharing the heterogeneous statistical strength across the local models while retaining their distinctive characteristics, our framework yields state-of-the-art performance. We also show that existing Bayesian PFLs are special cases of our framework.

Harnessing Heterogeneous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Mahendra Singh Thapa'
Rui Li
Abstract
Personalized federated learning (PFL) based on Bayesian approach tackle the challenges from sta- tistical heterogeneity of client data by comput- ing personalized posterior distribution over the parameters of each client's local model and con- structing global distribution by aggregating the parameters of these personalized posteriors. How- ever, the heuristic aggregation methods introduce strong biases and result in global models with poor generalization. We thus propose a novel hi- erarchical Bayesian inference framework for PFL by specifying a conjugate hyper-prior over the parameters of the personalized posteriors. This allows us to jointly compute a global posterior distribution for aggregation and the personalized ones at local level. This hierarchical Bayesian inference framework achieves elegant balance be- tween local personalization and global model ro- bustness. Extensive empirical study shows that by effectively sharing the heterogeneous statistical strength across the local models while retaining their distinctive characteristics, our framework yields state-of-the-art performance. We also show that existing Bayesian PFLs are special cases of our framework.
Samples Sample Mean
2
1
0
10
0.0
0.5
1.0 Variance
1.5
2.0
Figure I. In Bayesian PFL, variance parameters of personalized distributions often follow skewed distributions. In such cases, the MLEMMAP estimate (e.g., arithmetic mean) is obviously poor estimate.
et al., 2024), and mobile applications (Hard al., 2018; Wu et al., 2020; Geng et al., 2024). However, FL. still faces sub- stantial challenges, mainly due to statistical heterogeneity and limited data availability on individual clients (Ye et al., 2023).
The Bayesian- based personalized FL (PFL) approaches fo- cusing learning personalized local models for individual clients perform Bayesian inference to derive personalized posterior distributions of local model parameters (Zhang et al., 2022; Liu et al., 2023; Bhatt et al., 2024; Makhija et al., 2024). The server then aggregates the parameters of these personalized posteriors to construct a global distribution. Specifically, the construction of the global distributions in these approaches typically involves arithmetic aggregation of personalized posterior parameters the global distribu- tion parameters (Zhang et al., 2022) approximation of the global distributions as the product of the personalized ones (Liiu et al., 2023), as shown Figure 2. The global dis- tributions are then transmitted back to all clients as a local regularization for next training iteration. These simple yet rigid aggregation schemes the global level introduce strong constraints that obseure unique characteristics dis- tinguishing the heterogeneous local data and lead to biased estimates of the global models with poor generalization, as illustrated in Figure 1.
1. Introduction
Federated lcarning (FL) emerges as a promising framework that enables collaborative model training across decentral- ized client devices while preserving data privacy (McMahan al., 2017). This approach demonstrates significant poten- tial in various real-world applications, such as healtheare (Yang al., 2021; Dayan al., 2021; Yang et al., 2024), recommendation systems (Muhammad et al., 2020; Yuan
'College of Computing and Information Sciences, Rochester Institute of Technology, Rochester, New York, USA. Correspon- dence to: Rui Li <xxlics @rit.edu...
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
There are Bayesian based FL approaches focusing on con- structing robust global models, such as computing global dis-

---

Harnessing Heterogeneous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
tributions by fitting standard distributions clients' point- estimated local models (Thorgeirsson & Gauterin, 2020; Chen & Chao, 2021), constructing point-estimated global models from local estimates (Yurochkin et al., 2019) through nonparametric Bayesian methods. Since these approaches only obtain generic global models without personalized dis- tributions for each client, they are less effective at capturing the diversity of non-IID data in real-world FL settings.
(Yurochkin al., 2019). However, these methods are less effective in capturing the inherent statistical heterogeneity present across clients.
2.2. Personalized FL
Personalized FL aims to introduce a personalized model for individual clients to tackle the heterogeneity that exists in the FL environment. Several perspectives have been ex- plored to achieve personalization, including meta-learning (Fallah et al., 2020; Acar et al., 2021b; Jcon et al., 2024), multi-task learning (Smith et al., 2017; Dinh et al., 2022), transfer learning (Yang et al.. 2019), prototype-based (Tan et al., 2022; Xu al., 2023), and clustering (Ghosh et al., 2020; Sattler et al., 2020). Additionally, hypernetworks have also been employed to predict client model parameters (Shamsian et al., 2021; Scott et al., 2024), while regulariza- tion techniques balance personalization with global model performance (T Dinh et al., 2020; Li et al., 2020; Acar et al., 2021a; Li et al., 2021). Some strategies split the model, learning shared feature extractor and personalized classi- fier, vice-versa (Arivazhagan et al., 2019; Liang et al., 2020; Collins et al., 2021). These methodologies yield point estimate solutions, which are susceptible to overfitting, par- ticularly in scenarios where clients possess limited data.
We thus propose novel hierarchical Bayesian inference framework for PFL, addressing these challenges by balanc- ing between local personalization and global model robust- ness, as demonstrated in Figure 2. Specifically, we define a conjugate hyper-priot prioo distribution over the parameters of the personalized posterior distributions, allowing us to perform joint Bayesian inference at both the global level and the lo- cal level. By computing higher level posterior distribution over the personalized posterior parameters on the server, instead of summarizing them through arithmetic means, our framework achieves an elegant compromise. Estimates based on the global-level posterior dependencies between the personalized distribution parameters are "shrunk" to- gether, so that the local models share the heterogeneous statistical strength of each other, while retaining their dis- tinctive characteristics. Our theoretical analysis shows that our framework recovers existing Bayesian PFL objectives when additional constraints are introduced, which subsumes them as special cases. Our extensive empirical analysis shows that the proposed hierarchical Bayesian inference PFL results in robust global and local models with superior performance.
2.3. Bayesian Personalized FL
Bayesian PFL leverage various Bayesian inference tech- niques to estimate the personalized posterior distribution. For instance, pFedBayes (Zhang et al., 2022) utilizes vari- ational inference, while others have explored Laplace ap- proximation (Liu et al., 2023) and Markov chain Monte Carlo (MCMC) methods (Kotelevskii et al., 2022; Bhatt et al., 2024). The formulation of global distributions in these Bayesian frameworks varies. Some methods, such as PFedBayes, employ arithmetic aggregation of personalized distribution parameters, while others approximate global distributions as the product of personalized one (Bhatt et al., 2024). In particular, PFedVEM (Zhu et al., 2023) claims the use of hierarchical Bayesian modeling, but they do not ad- here to the principle due to missing hyper-priors over the pa- rameters of personalized posteriors. Studies utilize Gaussian Processes for personalization (Yin et al., 2020; Achituve et al., 2021; Tang et al., 2022) with mixed-effeee modeling to characterize feature extractors and personalized classifier layers (Kotelevskii al., 2022) and functional space prior to enhancing collaboration (Makhija et al., 2024).
2. Related Works
2.1. FL and Bayesian FL
In the seminal work of FL. FedAvg (McMahan et al., 2017) trains a local model on clients, aggregates them on the server considering only model parameters to construct a global model, and broadcasts this global model to all clients for subsequent training. Subsequent research efforts have fo- cused on improving convergence (Karimireddy et al., 2020), ensuring privacy guarantees (Agarwal et al., 2018), enhanc- ing communication efficiency (Reisizadeh et al., 2020), and refining client aggregation (Zhu et al., 2021). Furthermore, several Bayesian-based approaches have been explored to obtain a global model by fitting Gaussian or Dirichlet dis- tributions clients' point-estimated local models to de- fine posterior distributions over global model parameters (Chen& & Chao, 2021; Thorgeirsson & Gauterin, 2020). An alternative approach focuses on defining the global pos- terior as the product of local posteriors under a uniform prior (Al-Shedivat et al., 2021). Additionally, researchers have investigated nonparametric Bayesian methods to con- struct point-estimated global models from local estimates
A hierarchical Bayesian model for FL is explored to de- fine the hyper-prior as a normal-inverse -nw-ese--eewhaatttt distribution over personalized posterior parameters (Kim & Hospedales, 2023). However, by fixing the personalized posterior co- variance to a identity matrix, this approach reduces the local models to Bayessia neural networks. Similarly. a
2

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hiirrrrciica Bayesian Inference
Server Global Posterior Distribution
Server
Server
Global Distribution
Global Distribution
Hyperprior
~ (0, )
Half (ormal))ra))
Aggregation
Aggregation
Aggregation
> A Tak/4k
KL (o(Wall ail p(Wil Mg: og))
Mg
K
MAi
Pg
K
M4g
>e -k-1
(pe of)
(pes Pi:)
(He)
(pt 7gk)
By
Client Personalized Distribution
Client &
Client ks
Personalized Distribution
Personalized Distribution
W4 ~ N(He o
W/ ~ NN(H,, of
Wh. ~ N (HL, of)
o = log(1 + exp(p=))
o (ux ny)²
(a) Ours
(b) PFcdBayes
(c) PFedVEM
Figure 2. Overview of our PFL framework based on hierarchical Bayesian inference (left) in comparison with existing Bayesian based PFLs such pFedBayes (Zhang al., 2022) (middle) and PFedVEM (Zhu et al., 2023) (right). As each client uploads its updated personalized distribution parameters (juu, Φ to the server and then downloads the aggregated global parameters (Mg. (Ag. our framework derives global posterior distribution p(Wv gg) to aggregate the personalized posteriors by defining hyper-prior parameters. Other methods hcuristically specify the parameters of global distribution as weighted sums of the personalized ones.
their
federated variational inference method proposes model distribution parameters of Bayesian neural networks as la- tent variables (Hassan et al., 2023). Yet, they only model the weights of the first hidden layers as random variables, while the remaining weights are MAP estimates.
terized by weights Wk. capture the underlying structure of its local data Dz. The likelihood is thus expressed as
p(D We)
II plulf(r;))) We)) (x.
(1)
We specify a multivariate Gaussian with diagonal covari- ance as the prior over the weights of each client's local model Wi as
3. Hierarchical Bayesian Inference for PFL
As shown in Figure 2, our hierarchical Bayesian inference PFL framework derives personalized posterior distribution over the parameters of each client's ' local model based on Bayesian neural network (BNN). We further define a hyper- prior distribution over the parameters of these personalized posteriors on the server. This hyper-prior enables us to ag- gregate these personalized posteriors by computing global posterior distribution when the clients upload their parame- ters the server at each iteration. Moreover, for the joint approximate inference, we present a stochastic variational inference scheme that split the computation of the overall objective into a server part for aggregation and a client part for retaining personalization.
P(Wal of =
(2)
where la and of denote the mean and variance of the Gaussian distribution.
3.2. Global Posterior Distribution Specification
hyper-prior distribution over the param- eters of the personalized posteriors. Notably, the parameters of client k's personalized posterior 0 = {M4 o are direct instantiations of the hyper-prior p(O,la) where denotes hyper-paramedt parameters.. Specifically, we model p(0 as the product of Gaussian distribution and Half-Normal distribution (Leone al., 1961):
We propose
3.1. baNed Personalized Posterior distribution
p( |ax) = p(1,, gg
We consider a FL system comprising a central server and K decentralized clients. this framework, each client employs a neural network f(-) its local model, parame-
= =N(H, o) Holf-Normal(r,lo)) (3)
where 0, = (M,, o3)) and a (af.02)
3

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hiirarchical Bayesian Inference
We adopt Half-Normal distribution as a prior over oj as it computationally stable and efficient (Röver et al., 2021).
through the global posterior by optimizing the second and the third terms of the overall objective in Equation (6) with respect to d:
The global posterior distribution can thus be written as
K Gg
K
E ((00g (d)
[KL ( O 0y)]]] (7)
p(O,, a) & (O))a) II II(Dumm)p((w)).. (4)
1=
+ KKL (q p(0 aa))
where W } denotes the local BNNs' weights of all the clients, and D = } are all the data. The ex- act computation of both the global and the local posterior distributions are intractable due to the non-linearity inher- ent in neural networks. Therefore, we employ variational inference (VI) for approximation.
For communication efficiency, we specify q(0g: as Dirac distribution centered on specific value 0, as q00gi の) = 8(0,, - 0,). This leads to a simplified server objective:
KL ([WWe: log (0yla)
Cg
3.3. Stochastic VI for the Hierarchical Bayesian PFL
(8)
We specify a variational distribution g(0g. W; o, θ) to ap- proximate the true posteriors as follows:
By incorporating the hyper-prior in Equation (3), we have:
K
K
q0 g: W; ②. Q = o(0gi Φ) II Ow)
(5)
bg
KL
(((W;;
k=1
X1 X111/1 112 +221laagll
where, C = [0 Here, q(0g: o) denotes the varia- tional distribution of the global posterior parameters and q(We; 0%) represents the personalized distribution over client kss local model weights.
where ||/s|1 = Σ pi.g ||² = E o.
igg and
M = [Wal- A x of and x2 o 1 az are regularization
coefficients.
We minimize the Kullback-Leibler (KLL) divergence be- tween the variational distribution and the true posterior with respect to the variational parameters (Ø) O):
Client Part of the Objective: In each client, we update the personalized posterior by minimizing its expected negative logeliiel over the local data while regularizing it with the global posterior. Thus, the k-th client part of the overall objective in Equation (6) is:
O argmin KL [g( W; (),,,(,,, o))
6
argmin 0
Eg( (Dg
L4
Eagees ..) lolog
(10)
+ Eq(e, (0) [KL
+ KL
+ KTL. (g(0g) (p)))la))
(6)
Specifically, we specify each client's local model a per- sonalized BNN. Thus its variational distribution is:
The resulting overall objective is known as evidence lower bound (ELBO) (Neal & Hinton, 1998; Jaakkola & Jordan, 2000; Blundell et al., 2015). The first term facilitates the adaptation of the local models to the corresponding local data. The second term aggregates the personalized posterior distributions through the global posterior distribution. The last term acts as a regularization for the global posterior parameters. This ELBO thus effectively balances personal- ization with global model generalization. We further split the computation of the overall objective into server part for aggregation and a client part for retaining personalization to avoid the potential communication overhead caused by frequently synchronizing personalized posterior parameters.
g(Wi: ((Wi: Pk: Mx.07)
M
(11)
= II WWW (..)
where b4 = (M and of = (o Ms.. and of denote the mean and variance of Gaussian distribution for the i-th weight of k-th client's local. To ensure that o takes non-negative values, we replace each o by another parameters Pt.:: such that a log(1 + exp(Pr.)))
In the context of FL systems, where each client data re- mains inaccessible to the central server, we employ a block- coordinate optimization approach (Wu & Lange, 2008; Di- akonikolas Orecchia, 2018) to optimize the objective
Server Part of the Objective: On the central server, we aggregate the parameters of the personalized posteriors

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
functions. The optimization process consists of alter- nating steps: first the server side, we optimize the global variational parameters while keeping the client-specific vari- ational parameters & fixed; second, on the client side, we update the personalized variational parameters @ for multi- ple rounds while fixing the & as a constant.
(i.e., A1 = A2 0), Equation 13 is simplifies to:
K
bgg Σ k=1
Bo) [-
(oog [0,)]
(14)
By setting the derivative of Ly with respect to 0g 0, we obtain pFedBayes' aggregation rule for pg::
4. Theoretical Analysis
K
We provide theoretical analysis for the statistical conver- gence of the personalized posterior distributions, and show the existing Bayesian PFL methods are special instances of our framework.
PRi
D''
EI(E : )2 + o
Furthermore, given additional constraint Mr Mag recover pFedBayes' aggregation rule for o²
4.1. Statistical Convergence
K
We analyze the convergence behavior of the personalized distributions for each client k: p(Wx/Op) towards the global mean jı = I which close to the pooled estimate W by the FedAvg method.
K
K
k=
WW.
Example I:: PFedVEM (Zhu et al., 2023). With the same constraint that X1 A2 0 and assuming each client equipped with specific global distribution variance ò 5 we can we-write Equation 13 as:
Theorem 4.1. Let the hyperparameters a to be fixed, and approximate the global posterior with pluy Wg 5(1, [)) -o2) for simplicity, then the convergence of the personalized distribution mean:
K
Sg = Σ Eq(We.o.) [- log loog p(Whlog))))
(15)
k=
W Buj + (1 -3x0
(12)
where 03:: ling ozg
where 3k- +
By setting the derivative of Lg" with respect to 0g to 0,
we have PFedVEM's aggregation rule for g
It indicates that the mean of the personalized distribution parameters: E [ lies in between the local MLE esti- mate 0₁ We and the global mean ça. The convergence of the personalized parameters towards the global mean is thus governed by 3x. There is larger convergence for clients with smaller measurement precision (e.g., due to smaller data sizes or noisy labels), since 3% + 1 as of + xo.
Ag = Σ fg
K
Ek= 79
where og and o²g (H4 it )2 o
As shown above, the arithmetic aggregation adopted by the cxisting Bayesian PFL methods compute the global dis- tribution by simply averaging the personalized posterior parameters. The samples from the averaged global distribu- tion limits their ability capture non-IlI characteristies of the local models, blurring the statistical heterogeneity of the local data. In contrast, the personalized posterior parameters in our proposed framework are sampled from the convex combination of the averaged distribution and each person- alized distribution's 's local estimates, as in Theorem 4. In Section 6.3, our ablation study shows that the arithmetic aggregation scheme setting A = A2 0 leads inferior performance.
4.2. Special Instances of our Hierarchical Bayesian Inference PFL
Our proposed framework is fully general for heuristic aggre- gations of existing Bayesian PFL methods. We can recover the existing methods' objectives as special cases by intro- ducing additional constraints to the server-side objective in Equation (9). We first expand Equation (9) as:
K
cg oz)
( [logg((W [g)]
(13)
Anll@a 112 + 2211@r111
5. Algorithm Implementation
Example I: PFydBayes (Zhang et al., 2022). Given the constraints that the regularization coefficients are to zero
We provide the pseudo-code of our algorithm along with a description in Appendix A.3.
5

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
PM
GM
PM
GM
100
100
100
100
Test Accuracy (%)
90
80
Test Accuracy (%)
80
70
40
70
10
Ours
Ours
60
PFedVEM
20
20
PayBaayes
Bayes
50
(0.01, 00.01,0.01) 0.01)
(0.5. 0.5)
u1.. 1.0)
2.0.2.00
10.01. 0.011
(0.s, 0.5)
(1.0. 1.01
(2.0,20)
(0.01, (0.01.0.01) 0.01)
(0.5, 0.5)
(1.0. 1.0)
2.01
(0.01. 10.01..00) 0.01)
(0.s, 0.5)
0.5.0..))
(1.0. 1.01
(2.0, 2.0,2.00
(2.0.
(a.b)
(a, b)
(a. b)
(a, b)
Sampling rate: 0.1
Sampling rate: 0.2
Figure 3. Average test accuracy (%) of Personalized Model (PM) and Global Model (GM) across varying statistical heterogeneity controlled by (a, b) and sampling rate (i.c., 0.1 and 0.2). Although the PM performance are comparable, our GM consistently outperforms other methods across varying levels of statistical heterogeneity and under different client participation rates.
Local BNN Optimization: To compute the local BNN for each client, outlined in Equation (10), utilize recent advancements as Dayes-by-Backpnog (Blundell et al., 2015), which formulates an unbiased estimator for gradients through re- ce-parameterization.
PM
GM
70
70
Test Test Accuracy (96
5.0
FedPer
50
Monte Carlo Approximation: We employ Monte Carlo
40
Perfed/vg
FodRep
40
FeddAg
FedProx
estimation approximate the expectation term in Equa- tion (10). The estimator is expressed as follows:
30
PFedME (PM)
30
PFedME (GM)
20
PFodBayes (PM)
20
PFodBayes (GM)
PFedVEM (PM)
PFadVEM (GM)
Ours (PM)
Ours (GM)
s Σ log (DaIM!!)
o.) [- pog
25 Communication Round
75
100
25
75
100
Cammunication Round
Figure 4. Test accuracy vs. communication rounds for 100 clients on CIFARIO. Our framework shows rapid convergence.
where we draw S samples from the personalized posterior distribution {(W;;
Base-Head Archttetture: Recent studies have highlighted the effectiveness of Base-Head architecture in PFL scenar- ios (Collins et al., 2021; Kotelevskii et al., 2022; Zhu et al., 2023). Following this paradigm, the global model is decom- posed into base and head components. The corresponding distribution parameters 0, of the global model partitioned into base (0gibase) and head (0 (0rhe)d)) compo- nents, such that 0, θ grbase @ θ @ghhead. where @ denotes a concatenation operation. Similarly, for each client kt the local model is decomposed into base and head components, with the corresponding personalized posterior distribution parameters expressed as: 0, Ok:base + 0khhedd- The base component facilitates the extraction of common feature representations across clients, while the head component enables client-specific personalization.
model parameters, regularized by global posterior distribu- tion. A subset of clients communicates these personalized posterior distribution parameters to the central server. The server aggregates these personalized posteriors through a global posterior distribution. Subsequently, the resulting global posterior parameters are broadcast to all clients, fa- cilitating the next training iteration. The code is available online
also
6. Experiments
6.1. Synthetic Experiment
In this experiment, we focus on assessing the performance of the global model across diverse statistical heterogene- ity scenarios of our proposed framework. We adopt (Li
Following (T Dinh et al., 2020; Zhu et al., 2023), at each communication round, clients perform local Bayesian infer- ence, deriving personalized posterior distribution over local
'htops://github http://hhh...co/////eee/aa//aa/a
6

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Table Average test. accuracy of PMs and test accuracy of GM FMNIST, CO and CTFAR100 datasets for 50, 100, and 200 clients. Test accuracies are presented in percentage and the best results are highlighted in bold.
50 Clients
100 Clients
200 Clients
Datasct
Method
(%)
GM (%)
PM (%)
GM (%)
PM (%)
GM (%)
Local BNN only
88.7 0.9
86.9 0.9
85.2 = 0.4
FedAvg
83.5 + 0.4
85.4 + 0.3
85.9 + 0.2
FedProx
84.8 + 0.5
86.3 + 0.2
86.5 + 0.1
Scaffold
85.6 + 0.2
85.4 + 0.1
84.6 + 0.0
FedPer
91.4
90.7
89.7 +.1
FMNIST
FedRep
0.1
9.17
8.1
PerFedAvg
8.2
8..6
8.2
pFedME
911
82.0
9.14
0.6
90.6
85.1
PFedBayes
91.9
$3.5
91.3
0.3
90.
84.4
PFcdVEM
91.8
83.9 0.3
0.14
9.77 0.2
86.2
Ours
92.2 0.3
86.1 + 0.4
0.3
85.9 + 0.2
90.8 -
85.2 0.1
Local BNN only
54.9
0.1
45.6 0.2
FedAvg
57.7 $7.7
59.4 0.6
59.2 - 0.3
FedProx
58.0 0.7
59.4 0.5
59.1 + 0.2
Scaffold
60.4 1 0.3
59.8 + 0.2
55.4 1 0.3
FedPer
72.7
6.
6.34
CIFARIO
FedRep
0.3
0.44
6.2
PerFedAvg
6.89
6,86
6.12
PFedME
7.1
16.6100
7.24
6.31
6.2
58.7 + 0.2
pFedBayes
0.3
1.
0.3
0.7
64.6
51.4
PFcdVEM
73.2 0.2
5..0
0.1
59.4 7.3
Ours
77.1 0.1
65.9
74.7
0.3
70.4 0.2
5..5
Local BNN only
39.5
31.5 1.0
26.2 - 0.4
FodAvg
0.5
49.4
44.7 0.5
FedProx
48.4
45.5 +.5
42,4 0.3
Scaffold
0.4
4.74
30.0 +.1
FedPer
4.7
0.7
9..6
CIFAR100
FedRep
50,9 0.9
4.6
30.5
PerFedAvg
52.1
4.5
40.1 0.3
pFedME
0.5
47.9
0.5
45.1
41.6
41.5 11.5 1.6
pFedBayes
49.6 0.3
42.5
4.2
4.3
40.1
37.4
pFedVEM
61.0
$2.8
0..2
52.3
51.1
49.2
Ours
+
56.0 + 0.3
62.2 + 0.3
55.4
56.4 + 0.6
52.4 0.4
et al., 2020) to generate synthetic data that utilize param- eters and b to modulate statistical heterogeneity across clients. Specifically, a controls the degree of divergence be- tween local models, while b regulates the variation in local data distributions among clients. The dataset is designed to train 10 class classifier on dimensional real-value data. Following (Li et al., 2020), we distribute the data to N 50 clients according to a power law distribution. We the experiments for 100 communication rounds. To sim- ulate real-world federated learning conditions with partial client participation, we measure performance across sam- pling rates of 0.1 and 0.2. For all methods, we employ a Bayesian Fully Connected network (Appendix A.4) for both global and local models, with local learning rate of 0.005, local batch size
We evaluate the performance of both Personalized Models (PMs) and Global Model (GM). The experimental results in Figure 3 demonstrate that our framework consistently out- performs the competing methods in GM performance across varying levels of statistical heterogeneity and under differ- ent client participation rates. Notably other methods' GMs exhibit significant performance degradation under high het- erogeneity conditions. This indicates that the computation of higher level posterior distribution over the parameters of the personalized posterior distribution on the server ef- fectively harness the heterogeneous statistical strength of the clients' non-Ill) data to construct a robust and stable global model. The performance PMs remains comparable across all approaches. This may attributed the rela- tive simplicity of the synthetic dataset, characterized low feature dimensionality and small data size, which enables all methods to effectively capture the underlying non-IID characteristics.
as local data size, and 10 local number of epochs. We tune the hyperparameters of pFedBayes and pFedVEM around their recommended values.

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
70
70
60
60
Test Accuracy (%)
Test Accuracy (%)
50
50
Local BNN only
Local BNN only
40
40
FedPer
FedPer
FedRep
FedRep
30
PerfedAvg
PerfedAvg
DFedME
pFedME
20
20
PFedBayes
pFedBayes
DFedVEM
pFedVEM
10
10
Ours
Ours
1000
2000
3000
0
500
1000
1500
2000
2500
Local data size
Local data size
(a) Client = 50
(b) Client = 100
Figure Test accuracy vs. local data size for 50 100 clients on CIFAR100. Our framework provides greater collaborative learning benefits for clients.
6.2. Performance Comparison on Real Data
configuration in (Zhu et al., 2023) (Appendix A.4). For each dataset, conduct grid search for hyperparameter op- timization for our proposed method using K 100 client and apply those configurations to K 50 and K 200 case. We report the average test accuracy (%) with corre- sponding standard error, computed over three independent runs, each with a different random seed. We conduct all the experiments using NVIDIA A5000 GPUs.
We examine various categories of statistical heterogeneity that exist in real-world FL scenarios label distribution skew: clients possess varying label distributions, label con- cept drift: feature distributions differ among clients, and data quantity disparity: clients have different amounts of data (Ye et al., 2023; Zhu et al., 2023). Following (Zhu et al., 2023), we assess performance on Mushion-MNISST (FMNIST) (Xiao et al., 2017), CIFARIO (Krizhevsky al., 2009), and CIFAR100 (Krizhevsky al., 2009) datasets. For label distribution skew, each client on FMNIST and CI- TORI0 datasets has unique local data of 5 labels out of 10. For label concept drift in CIFAR 100, the task is superclass predictions. We compare the performance of our approach with the following FL. frameworks: FedAvg (McMahan et al., 2017), FedProx (Li et al., 2020), Scaffold (Karim- ireddy et al.. 2020), FedPer (Arivazhagan al., 2019), Fe- dRep (Collins et al., 2021), PerFedAvg (Fallah al., 2020), pFedME (T Dinh et al., 2020), pFedBayes (Zhang et al., 2022), PFM (7hu et al., 2023), and Local BNN only, which refers to BNN optimized independently at each client.
6.2.1. PERFORMANCE COMPARISON
Table presents a comprehensive performance across three datasets: FMNIST, CIFAR10 10 and CIFAR100, with vary- ing number of clients (50, 100, 200). Our Global Model (GM) consistently outperforms existing approaches. On the CIFARIO dataset, our GM outperforms competing mod- els by 5.5%, 3.7% for 50 and 100 clients, respectively, and achieves comparable performance for 200 clients. Our GM's performance the CIFARI 100 dataset is consistently supe- rior, surpassing other models by 3.2%, 3. 3.1%, and 3.2% for 50, 100, and 200 clients, respectively. These results indicate that our framework effectively shares the heterogeneous statistical strength across local models via a hierarchical setup. For the FMNIST dataset, especially with 100 and 200 clients, our GM maintains performance comparable to state-of-the-ant approaches, likely due the relative sim- plicity of the dataset and the limited presence of non-IID characteristics. Our Personalized Model (PM) demonstrates superior performance compared to the competing methods. On the FMNIST dataset, our approach exhibits superior per- formance across all client configurations. On the CIFAR 10 dataset, our PM consistently outperforms competing algo-
We run the experiments for 100 communication rounds. To emulate real-world FI. conditions characterized by partial client participation, each client has a probability of 0.1 to communicate its local model parameters back to the server during each round of communication following a binomial distribution. The experimental setup encompasses varying numbers of clients, specifically 50, 100, and 200. For model architectures, we employ Bayesian MLP with one hidden layer for FMNIST, a LeNet-like Bayesian CNN for Cifar10, and 6-layer Bayesian CNN for CIFAR- 100 similar to the
8

---

Harnessing Heterogeneous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Table Average test accuracy of PM and GM on CTFAR 100 dataset of method using different global model regularization cnefficients. Test accuracies are presented in percentage and the best results are highlighted in bold.
Regularization Coefficients
50 Clients
100 Clients
200 Clients
A2)
PM (%)
GM (%)
PM (%)
GM (%)
PM (%)
GM (%)
(0, 0)
61.9 + 1.1
53.1 = 0.2
59.4 0.4
52.9 + 0.2
55.3 + 0.2
51.1 = 0.5
(1, 1)
63.2 + 1.1
54.7 = 0.4
60.7 0.3
0.4
56.2 0.3
51.8 = 0.4
(0, 5)
61.7 + 0.6
53.8 = 0.2
0.4
53.8
0.2
55.9 0.1
52.4 = 0.7
(5, 0)
65.4 + 1.2
54.6 = 0,4
60.7
0.3
0.3
53.9
48.1 = 0.8
(5
66.6 0.4
56.0
0.3
62.2
20.20033 0.3
5.1
56.4
52.4 0.4
(10, 10)
66.6 = 1.3
51.9 = 0.7
62.2 - 0.2
3.7
0.4
49.2 = 1.6
rithms, with performance gains of 3.9%, 2.8%, and 0.3% for 50, 100, and 200 clients, respectively. The performance gains are even more pronounced on the CIFARIO0 dataset, where our PM surpasses competing algorithms by substan- tial margins of 5.6%, 6.0%, and 5.3% for 50, 100, and 200 clients, respectively. This demonstrates the ability of our method to effectively retain distinctive characteristics of local data via enabling client-level model personalization.
indicates that the hyper-prior over both Bg and og plays crucial role in enhancing the model performance. Fur- thermore, excessive regularization (e.g., (10, 10)) negatively impacts the model performance, particularly for the global model. This suggests that overly stringent regularization can potentially constrain the model's learning capacity.
We investigate the impact of updating order of local model's base and head components in Appendix A.5. Consistent with (Collins et al., 2021; Zhu et al., 2023), our findings indicate that updating the head component prior to the base yields the best performance. We also demonstrate the scalability of our proposed method to large number of clients and the generalization to more complex datasets, along with the evaluation of uncertainty quantification in Appendix A.9.
Moreover, Figure 4 shows the convergence behavior of vari- ous algorithms on CIFARIU 10 for 100 clients. Our framework demonstrates rapid convergence, achieving competitive per- formance at approximately half the total communication rounds required by the baseline methods. This efficiency is crucial in FL. where minimizing communication rounds is important. Similar convergence patterns observed for scenarios involving 50 and 200 clients (Appendix A.7).
Furthermore, Table shows the overall performance of PFL methods consistently surpasses that of Local BNN only approach across all experimental settings. We further inves- tigate the collaborative learning benefits across clients with diverse local data sizes. Specifically, we perform an exper- iment using CIFARI00 across 50 and 100 clients and plot test accuracy over local data size in Figure 5. It shows that our framework provides greater benefits to clients compared to the competing approaches and incentivize participation from clients for the collaborative training process.
7. Conclusion
Our hierarchical Bayesian inference PFL framework speci- fies a hyper-prior over the parameters of personalized poste- riors. This enables us to jointly compute global posterior for aggregation and local posteriors for personalization. Re- sults shows by sharing heterogeneous statistical strength across clients while retaining their distinctive characteris- tics, our framework yields state-of-the-ar performance. Our hierarchical Bayesian approach for PFL offers a principled framework that enables effective generalization in low-data regimes, quantifies uncertainty, and enhances robustness (Cao et al., 2023).
6.3. Ablation Study
We conduct an ablation study to investigate the impact of regularization on model performance. For this, we perform experimentation using CIFAR100 across 50, 100, and 200 clients. Table 2 shows the model's performance without regularization (Λi, A²) (0, 0) consistently underperforms compared to regularized configurations. It suggests that hyper-prior over personalized posterior parameters plays an important role in enhancing both PM and GM generaliza- tion. The best performance achieved with regularization coefficients (A,, A2) (5.5), 5). Importantly, with either X or A set to O, the model's performance degrades. This
A Acknovledgements
This work is supported by the National Science Foundation under NSF award No.2045804 and National Institutes of Health under NIH award No.1R35GM1 IR35GM156653-00 We ac- knowledge the Research Computing at Rochester Institute of Technology (RIT, 2024) for the computational resources.
9

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hiirrrcciicl Bayesian Inference
Impact Statement
of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 7233-7242, 2023.
Federated learning enables collaborative model training across decentralized client devices while preserving data pri- vacy, and demonstrates significant potential in various real- world applications. Our framework effectively addresses the challenges that federated learning systems face in general due to statistical heterogeneity and limited data availability on individual clients.
Chen, H.-Y. and Chao, W.-L. Fed [be{ Making bayesian
model ensemble applicable to federated learning. In International Conference on Learning Representations, 2021. URL https: https:://ppeneeiieeeese http///ppeeeeeeeeeeeeeeeeeeeeeee net forum?
id=dgtpE6gK33
Collins, L., Hassani, H., Mokhtari, A., and Shakkottai, S. Exploiting shared representations for personalized feder- ated learning. In International conference on machine PMLR, 2021.
References
Acar, D. A. E., Zhao, Y., Matas, R., Mattina, M., What- mough, P., and Saligrama, Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021a. URL https: // //openreviie net forum? 7dd-B744M666766
learning. pp.
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al. Federated learning for predicting clinical outcomes in patients with covid-19, 19. Nature medicine, 27 (10::133--1443, 2021.
Acar, D. A. E., Zhao, Y., Zhu, R., Matas, R., Mattina, M., Whatmough, P., and Saligrama, V. Debiasing model updates for improving personalized federated training. In International conference on machine learning, pp. 21-31. PMLR. 2021b.
Diakonikolas, J. and Orecchia, L. Alternating randomized block coordinate descent. In International Conference on Machine Learning, pp. 1224-1232. PMLR, 2018.
Achituve, I., Shamsian, A., Navon, A., Chechik, G., and Fetaya, E. Personalized federated learning with gaussian processes. Advances in Neural Information Processing Systems, 34:8392-8400 2021.
Dinh, C. T., Vu, T., Tran, N. H., Dao, M. N., and Zhang, H. A new look and convergence rate of federated multitask learning with laplacian regularization. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Agarwal, N., Suresh, A. T., Yu, X. X.. Kumar, S., and McMahan, B. cpsgd: Communication--ffiieeet and differentially-priivat distributed sgd. Advances in Neural Information Processing Systems, 31, 2018.
Fallah, A., Mokhtari, A., and Ozdaglar, A. Personalized federated learning with theoretical guarantees: A model- agnostic meta-learning approach. Advances in neural information processing systems, 1-:357--35683 2020.
Al-Shedivat, M., Gillenwater. J.. Xing, E., and Ros- tamizadeh, Federated learning via posterior averag- ing: A new perspective and practical algorithms. In International Conference Learning Representations, 2021. URL https: /tttps///pen..eviee...ee.
Geng, J., Tang, B., Zhang, B., Shao, J., and Luo, B. Fedcam- pus: A real-world privacy-preservee mobile application for smart campus via federated learning & analytics. In Proceedings of the Twenty-Fifth International Symposium on Theory, Algorithmic Foundations, and Protocol De- sign for Mobile Networks and Mobile Computing, pp. 377-378, 2024.
Arivazhagan, M. G., Aggarwal, V., Singh, A. K., and Choud- hary, S. Federated learning with personalization layers. 200888. 2019.
Ghosh, A., Chung, J., Yin, D.,, and Ramchandran, K. efficient framework for clustered federated learning. Ad- vances in Neural Information Processing Systems, 33: 19586-19597, 2020.
arXiv preprint
Bhatt, S., Gupta, A., and Rai, P. Federated learning with uncertainty via distilled predictive distributions. In Asian Conference on Machine Learning, pp. 153-168. PMLR, 2024.
Hahn, S.-J., Jeong, M., and Lee, Connecting low-loss subspace for personalized federated learning. In Proceed- ings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 505-515, 2022.
Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In International conference machine learning, pp. 1613-1622. PMLR, 2015.
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., and Ramage, D. Federated learning for mobile keyboard prediction. arXiv preprint 18/ 1.03604, 2018.
Cao, L., Chen, H., Fan, X., Gama, J.. Ong, Y.-S., and Kumar,
V. Bayesian federated learning: survey. In Proceedings
10

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Hassan, C., Salomone, R., and Mengersen, K. Federated variational inference methods for structured latent vari- able models. arXiv preprint arXiv: 03314, 2023.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-effiiie learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Jaakkola, T. S. and Jordan, M. I. Bayesian parameter estima- tion via variational methods. Statistics and Computing, 10:25-37, 2000.
Muhammad, K., Wang, Q. O ReRill D., Tragos, E., Smyth, B., Hurley, N., Geraci, J., and Lawlor, A. Fedfast: Going beyond average for faster training of fed- erated recommender systems. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1234-1242, 2020.
Jeon, I., Hong, M., Yun, J., and Kim, G. Federated learn- ing via meta-variational dropout. Advances in Neural Information Processing Systems, 36, 2024.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averag- ing for federated learning. In International conference on machine learning, pp. 1132-5143, PMLR, 2020.
Nacini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelli- gence, volume 29, 2015.
Kim, M. and Hospedales, Fedhb: Hierarchical bayesian federated learning. arXiv preprint EXXi:2305.04999 2023.
Neal, R. M. and Hinton, G. E. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pp. 355-368. Springer, 1998.
Kotelevskii, N., Vono, M., Durmus, A., and Moulines, E. Fedpop: A bayesian approach for personalised federated learning. Advances in Neural Information Processing Systems, 35:8687-8701, 2022.
Oh, J., Kim, S., and Yun, S.-Y. FedBABU: Toward hanced representation for federated image classification. In International Conference on Learning Representations, 2022. URL https:: opwnrevieww net/ forum? Yd=
Krizhevsky, et al. Learning multiple layers of features from tiny images. 2009.
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, R. Fedpaq: A communication eefifccctti federated learning method with periodic averaging and quantization. In International conference artificial intelligence and statistics, pp. 2021-2031. PMLR, 2020.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Leone, F. C., Nelson, L. S., and Nottingham, R. The folded normal distribution. Technometrics, 3(4):543-550, 1961.
RIT. Research computing services, 2024. URL https: / //www. rit edu/
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-4505 2020.
Röver, C., Bender, R., Dias, S., Schmid, C. H., Schmidli, H., Sturtz, S., Weber, S., and Friede, T. On weakly infor- mative prior distributions for the heterogeneity parameter in bayesian random-effects meta-analysis. Research Syn- thesis Methods, 12(4):448-47) 2021.
Li, T., Hu. S., Beirami, A., and Smith, V. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pp. 6357- 6368. PMLR, 2021.
Sattler, F., Müller, K.-R., and Samek, W. Clustered federated learning: Model-agnostic distributed multitask optimiza- tion under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8):371--3722, 2020.
Liang, P. P., Liu, T., Ziyin, L., Salakhutdinov, R., and Morency, L.-P. Think locally, act globally: Federated learning with local and global representations. arXiv preprint ar XrXiv:2001.01523. 2020.
Scott, J., Zakerinia, H., and Lampert, C. H. PeFLL: Per- sonalized federated learning by learning learn. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: openrevien.eeeeeeeee htt.siewwweeeeeeee.......... forum? Beruniid-MMYYiwDDR0
Liu, L.,, Jiang, X., Zheng, F., Chen, H., Qi, G.-J,, Huang, H.,, and Shao, L. A bayesian federated learning framework with online laplace approximation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Makhija, D., Ghosh, J., and Ho, N. A bayesian approach for personalized federated learning in heterogeneous settings, 2024. URL https:/ httpp/ppeevveeeeeeee wwt.eewwwwwwweeeeeeeeee..e.eeeee forum? 1d=0hTzuwz06Q.
Shamsian, A., Navon, A., Fetaya, E., and Chechik, G. Per- sonalized federated learning using hypernetwrkss. In International Conference on Machine Learning, pp. 9489- 9502. PMLR, 2021.

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Smith, V., Chiang, C.-K., Sanjabi, M., and Talwalkar, A. S. Federated multi-task learning. Advances in neural infor- mation processing systems, 30, 2017.
Yang, T., Yu, X., McKeown, M. J., Wang. Z. J., et al. When federated learning meets medical image analysis: A sys- tematic review with challenges and solutions. APSIPA Transactions on Signal and Information Processing, 13 (1), 2024.
Dinh, C., Tran, N., and Nguyen, Personalized federated learning with moreau envelopes. Advances in neural information processing systems, 33:21394-2144 2020.
Ye, M., Fang, X., Du, B., Yuen, P. C., and Tao, D. Hetero- geneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56(3):1-44, 2023.
Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q.. Jiang, J., and Zhang, C. Fedproto: Federated prototype learning across heterogeneous clients. Proceedings the the AAA/ Confer- ence on Artificial Intelligence, volume 36, pp. 8432-8440, 2022.
Yin, F., Lin, Z., Kong, Q., Xu, Y., Li, D., Theodoridis, S., and Cui, S. R. Fedloc: Federated learning framework for data-driven cooperative localization and location data processing. JEEE Open Journal of Signal Processing, 1: 187-215, 2020.
Tang, M., Ning, X., Wang, Y., Sun, J., Wang, Y., Li, H., and Chen, Y. Fedcor: Correlation-based active client selection strategy for heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 10102-10111, 2022.
Yuan, W. Qu, L.., Cui, L., Tong, Y., Zhou, X., and Yin, H. Hetefedrec: Federated recommender systems with model heterogeneity. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pp. 1324-1337. IEEE, 2024.
Thorgeirsson, A. T. and Gauterin, F. Probabilistic predic- tions with federated learning. Entropy, 23(1):41, 2020.
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang, N., and Khazaeni, Y. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pp. 7252-7261. PMLR, 2019.
Van Rooyen, B., Menon, A., and Williamson, R. C. Learn- ing with symmetric label noise: The importance of being unhinged. Advances neural information processing systems, 28, 2015.
Wu, T. T. and Lange, K. Coordinate descent algorithms for lasso penalized regression. The Annals of Ap- plied Statistics, 2(1):224 - 244, 2008. doi: 10.1214/ 07-AOAS147. URL https:/ http::///....og......30. wrt./o..g/111..... 1214/ 07 07-A0AS147.
Zhang, Li, Y., Li, W., Guo, K., and Shao, Y. Personal- ized federated learning via variational bayesian inference. International Conference on Machine Learning, pp.
26293-26310. PMLR, 2022.
Zhu, J. Ma, X., and Blaschko, M. B. Confidence-aware per- sonalized federated learning via variational expectation maximization. In Proceedings of the IEEE/CVÉ Confer- ence on Computer Vision and Pattern Recognition, pp. 24542-24551, 2023.
Wu, X., Liang, Z., and Wang, J. Fedmed: A federated learning framework for language modeling. Sensors, 20 (14):4048, 2020.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv:178..077 2017.
Zhu, Z., Hong, J., and Zhou, J. Data-free knowledge distil- lation for heterogeneous federated learning. In Interna- tional conference on machine learning, pp. 12889. PMLR, 2021.
Xu, J.. Tong, X., and Huang, S.-L. Personalized fed- erated learning with feature alignment and classifier collaboration. In The Eleventh International Confer- ence on Learning Representations, 2023. URL ht tts::: //openrevien net/ forum? 1d=
Yang, Q., Liu, Y., Chen, T., and Tong, Y. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 1092):1-19, 2019.
Yang, Q. Zhang, J., Hao, W., Spell, G. P., and Carin, L. Flop: Federated learning on medical datasets using partial networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3845-3853, 2021.
12

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
A. Appendix
A.I. Stochastic VI for the Hierarchical Bayesian PFL
We minimize the Kullback-Leibler (KIL) divergence between the variational distribution q(O,, W; Ø, () and true posterior p WID, ai) with respect to the variational parameters (Ø, )):
argmin KL (g(6,, W: 0)lp( W(D,a.)))
6.
By the definition of KL divergence:
log g00 W; 6. θ) p(0g. a.)
KL [g(0 W; 0)lp((0,, WID, o)] = Wio
Using Bayes' theorem:
W(D,a) (W
p(D(a)
Thus, the KL divergence becomes:
Wis.o) olog ( 6; @) - (lg)((D)) - logg(())g)) - log log
+ log (D]a))
Since p(D(a) is independent of o and θ minimizing the KL divergence is equivalent to minimizing the following ELBO:
L = Egte,, W. log P(D))) log)(W))g)) - log p(0g + log (0 g: W; db O)]
We specify a variational distribution (0 g. W; c. ) to approximate the true posteriors as follows:
log ( g: W; Φ. o log ( x) + log g(W; O)
where (00 gi ②) denotes the variational distribution of the global posterior parameters and g(W; Q) represents the personal- ized posterior distribution over client local model weights.
Thus, the ELBO becomes:
[ Wo..e) logp(D(W)) log log logg((,)a) log (MV; θ)]
F. logp(D(() log WW: +) p(W10,)
+ log {0g; p(0, ax)
As clients are conditionally independent given the common prior 0g, the ELBO becomes:
C 1-lg + Ey(0, (Eg flog (We; 0x), p(W, 0,,) [1] + Eg(o, log g(0g: の) p(0, (a)
+ (x(O):)) (x) KL (
A.2. Statistical Convergence
After computing the approximate global posterior in Equation (4) via stochastic variational inference (i.c., Section 3.3). we analyze the convergence of the personalized distribution for each client A: p(W, (x), where O₂ {1 o are the parameters of the client k 'ss personalized posterior, as in Section 3.2, and W3 denotes the weights of the local model (i.c., Bayesian neural networks).
Specifically, analyze the convergence behavior of the personalized distributions towards the global mean j E WW) which is close to the pooled estimate W. Let the hyperparameters a to be fixed, and we adopt the approximation the
13

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
global posterior p( W) 8(14, ()) o²2) for simplicity, then the marginal distribution of the personalized distribution parameters is:
p(0+ / p( My. 0g)p(0, W)d0,
(16)
(17)
note that 0,- = (Mg. gg)
Meanwhile, the above marginal posterior be expressed as
p(0 p(0x|Wx p. 0²) x p(W))(x) p(Oxla,
(18)
= - (N k4, σ²)
(19)
(0 - 02)²
= exp(
eepp
(20)
o
Ø²
= exp(-(
1
00
+
02
+
p
0
0
2
n²
))
(21)
20
20
ok
σ
20 k
20
where 02 Wi denotes the local MLE estimate.
Since the marginal posterior p( WW, μt, o²) [(On|E V Var[Ox W]) is Gaussian due to conjugacy, by matching the coefficients the quadratic and linear terms the two expressions, we have the variance of the marginal distribution as
1
1
+
(22)
aa W]
2
2
k
0
Var
W|
(23)
k
+
and the mean of the marginal distribution as
W
0
+
(24)
Var
0 W
2
k
0x
n²
W
Var[Ou|\W]
(25)
0k
3xj + (1 - 3x)02
(26)
of t,ta"
where 3k
It indicates that the mean of the personalized distribution parameters: E|0 [W] lies in between the local MLE estimate 0, Wi and the global mean μ.. The convergence of the personalized parameters towards the global mean is governed by 3g. Thus, we see that there is larger convergence for clients with smaller measurement precision (e.g., due to smaller data sizes or noisy labels), since B4 + as o¹ 00.
A.3. Algorithmic Description
In our proposed framework, clients perform local Bayesian inference at each communication round, deriving personalized posterior distribution over local model parameters. A subset of clients communicates these parameters to the central server. The central server then aggregates these personalized posteriors to construct a global posterior, which regularized by a hyper-prior. Subsequently, the server broadcasts the latest global posterior to all elients for the next training iteration. The full procedure of our proposed works is shown Algorithm 1.
A.4. Model Architecture
The Model architectures for Synthetic, FMNIST, CIFARIO, and CIFAR100 datasets used in this proposed work are shown in Figure We employ a Bayesian Fully Connected network for synthetic data, a Bayesian ML. with one hidden layer
14

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
for FMNIST, LeNet-like Bayesian CNN for Cifarl0, and 6-layer Bayesian CNN for CIFAR-100. Each architecture incorporates ReLU activation functions following Bayesian Fully Connected networks and Bayesian Convolutional Layer. All models are structurally decomposed into Base and Head components, with the Head component comprising only the last layer, while the Base component encompasses all preceding layers. Notably, as the synthetic data model consists of Bayesian Fully Connected layers only, it doesn't have a Base component.
Model Output
Head
100 x 10 BFC
Adaptive Avg Pool
Model Output
3 3 100 BCONV
x 2 Max Pool
Head
84 x 10 BFC
3 3 @ 256 BCONV
120 84 BFC
2 Max Pool
800 x 120 BFC
3 @ 128 BCONV
Model Output
2 Max Pool
2 Max Pool
5 x @ 32 BCONV
3 3 @ 64 BCONV
Model Output
Head
100 10 BFC
2 Max Pool
Max Pool
Head
64 x 10 BFC
Base
784 x 100 BFC
Base
5 5 @ 16 BOONV
Base
3 3 @ 32 BCONV
Input Data
Input Data
Input Data
Input Data
Synthetic Data
FMNIST
CIFAR10
CIFAR100
Figure 6. Model architectures for Syathetic, FMNIST, CIFARIO and CIFARI00 datasets. BFC refers to Bayesian Fully Connected Layer and BCONV refers Bayesian Convolutional Layer.
PM
GM
PM
70
70
70
Test Accuracy (%)
Test Accuracy (9%)
50
FedPer
50
FadPer
50
FedRep
Fedfvg
FedRep
FedAwg
40
PerFedligg
FedProx
PerfedAwg
40
30
PFedME (PM)
PFadME (GM)
PFodME (PM)
30
pRadME (GM)
20
PFedVEM
PeddBayes (PM)
20
PFedBayes (GM)
PFadVEM
20
DFddaayes (PM)
(PM)
20
pFedBayes (GM)
(GM)
Ours (PN)
Ours (GM)
Ours (PM)
Ours (GM)
100
100
Communication Round
Communication Round
Communication Round
Communication Round
(a) Client 50
(b) Client 200
Figure 7. Test accuracy vs. communication rounds for 50 and 100 clients on CIFAR10.
A.5. Base-Head Updating Order
We investigate the impact different updating orders of the local model's base and head components. Specifically, we consider three updating orders: updating the base component only (Oh et al., 2022), updating the base followed by the head,
15

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Table Average test accuracy of PM and GM CI dataset of method using different update orders. Test accuracies
presented in percentage and the best results are highlighted in bold.
50 Clients
100 Clients
200 Clients
Update Order
Sharing Component
PM (%)
GM (%)
PM (%)
GM (%)
(%)
GM (%)
Base only
Base only
76.8 0.2
72.9 + 0.2
68.3 0.4
Base + Head
Both base & head
76.7 0.4
65.1 - 0.5
72.6 + 0.8
60.6 0.6
68.3 = 0.4
57.3 = 0.6
Head + Base
Both base & head
77.1 + 0.1
65.9 + 0.9
74.7 + 0.3
63.8 + 0.3
70.4 + 0.2
59.5 + 0.7
and updating the head followed by the base (Collins et al., 2021; Zhu et al., 2023). We perform experimentation using CIFARIO across 50, 100 and 200 clients. The experimental results in Table 3 indicate that the strategy of updating the head followed by the base consistently yields superior performance. This updating order facilitates initial adaptation of the head components to the client's ' local data, which is subsequently followed by refinement of shared feature representation through updates to the base components.
A.6. Hyperparameters
We list all the hyperparameters of our method for the Synthetic, FMNIST, CIFARIO, and CIFAR100 datasets in Table 4. Our initial hyperparameters configuration is based on the recommendation provided by pFedVEM (Zhu et al., 2023). Subsequently, we leverage grid search to finetune these hyperparameters for our proposed method. Compared to baseline methods, our proposed approach incurs additional computational overhead of tuning the regularization coefficients X and A via grid search. Our ablation study, in Section 6.3, demonstrate that these regularization coefficients are critical to the superior performance for Bayesian personalized federated learning (pFL.) in general. Disabling them leads to substantial degradation in performance, highlighting the major performance bottleneck of the prior approaches.
Table Hyperparameters of proposed method for various datasets. For the synthetic data, which consists only of the head component, base component hyperparameters are not applicable (denoted by -).
Hyperparameter
Synthetic Data
FMNIST
CIFAR1O
CIFAR100
Local Batch size base
50
50
Local Batch size for head
Local data size Adam
Local data size
Local data size
Local data size
Local optimizer
Adam
Adam
Adam
Local learning rate for base
0.001
0.001
0.001
Local learning rate for head
0.005
0.001
0.001
0.001
Local epochs base
4
Local epochs for head
10
10
20
Global learning rate for base
0.01
0.01
0.01
Global learning rate for head
0.01
0.01
0.01
0.01
Global epochs for base
5
5
Global epochs for head Global optimizer A
10
5
5
Adam
Adam
Adam
Adam
5
5
5
5
5
Local KLL coefficient for base
10
10-6
10-6
Local KL coefficient for head
1
1
A.7. Convergence Behavior
Figure 7 show the convergence behavior of various algorithms on the CIFARIO dataset for 50 and 200 clients, respectively. Our proposed method demonstrates rapid convergence for 50 clients, achieving the competitive performance of PM and GM at approximately one-fourth and half of the communication rounds needed by the baseline approaches. In the case of 200 clients, our method's convergence rate is comparable to other competitive approaches.
16

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
A.8. Computational Cost
We measure the average local training time per communication round across three configurations: non-Bayesian (McMahan et al., 2017), non-Bayesian base with Bayesian head (Zhu et al., 2023), and full-Bayesing approaches (Zhang et al., 2022). Our proposed method also employs full-Bayesian framework. For this, perform experiment using CIFARIO, with all 100 clients participating in each communication round. As shown in Table 5, full-Bayesian methods incur high computation costs, leading to a longer local training time per communication round.
Table 5. Average local training time (in seconds) per communication round on CIFARIO with all 100 clients available
Method
Local Training Time (seconds)
non -on-Bayesian
21.42 + 1.36
non-Bayesian base with Bayesian head full-Bayesian
43.14 + 3.22
90.38 + 4.50
A.9. Additional Experiments
1.9.1. SCALABILITY UNDER LARGE NUMBER OF CLIENTS
To evaluate the scalability of our proposed methods under significantly large number of clients, we conduct experiments on the CIFARIO dataset with 1000 clients, where each client has unique local data labels out of 10. As shown Table 6, our methods scale well with a large number of clients.
Table Average test accuracy and GM on CIFARIO dataset 1000 clients. Test accuracies are presented in percentage and the best results are highlighted in bold.
Method
PM (%)
GM (%)
FedAvg
48.7 + 1.1
FFcdVFM FM
59.0 0.4
48.6 + 0.9
Ours
61.0 0.3
52.3 1 0.4
A.9.2. GENERALIZATION UNDER COMPLEX DATASET
To assess the generalization capability of our proposed method on a more complex dataset with a large number of categories, conduct experiments the Tiay-ImageNet dataset (Le & Yang, 2015). The experimental setup includes 50 clients, each with local data consisting of 50 labels out of 200. As presented in Table 7, proposed framework achieves the best performance.
Table Average test accuracy of PM and GM on Tiny- dataset for 50 clients. Test accuracies presented in percentage and the best results are highlighted in bold.
Method
(%)
GM (%)
FedAvg
10.1 +
PFedVEM
33.5 0.2
18.5 +
Ours
42.4 + 0.1
25.7 + 0.1
A.9.3. UNCERTAINTY QUANTIFICATION
To demonstrate the effectiveness of the Bayesian framework in uncertainty quantification, we evaluate predictive uncertainty on the CIFARIO dataset. We adopt (Hahn et al., 2022) to simulate noisy labels scenario by applying symmetric flipping ( (Van Rooyen et al., 2015) with a noise ratio 0.2. We report the test Expected Calibration Error (ECE) (Naeini et al., 2015)
and test accuracy. As shown in Table 8, our proposed method's 's uncertainty quantification comparable with PFedVEM in the Personalized Model setting, while consistently achieving higher accuracy across all experimental configurations.
17

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Table Average Expected Calibration Error (ECE) and accuracy (Ace) of PM and GM CIFAR 10 dataset under noisy labels. Test accuracies are presented in percentage and the best results are highlighted in bold.
50 Clients
100 Clients
Method
PM: ECE (Acc)
GM: ECE (Ace)
PM: ECE (Acc)
GM: ECE (Acc)
FedAvg
0.37 0.06 (36.8 0.9)
0.35 = 0.08 (37.9 2.5)
PFedVEM
0.13 - 0.01 (70.1 = 0.4)
0.19 = 0.01 (46.1 2.3)
0.17 + 0.00 (68.0 0.4)
0.22 = 0.00 (49.6 2.9)
Ours
0.15 + 0.00 (72.6 + 0.4)
0.28 = 0.00 (59.8 + 0.3)
0.16 + 0.01 (70.3 0.3)
0.26 + 0.00 (57.6 + 0.7)
A.9.4. COMPARISON WITH NoN-FL SETUP
We compare the performance of our proposed method against non-federated learning (non-FL) centralized setup, imple- mented using deterministic neural network trained the entire dataset. As shown in Table 9, the performance of the centralized setup does not necessarily case of personalized model. This may be attributed to the centralized model's ' need to generalize across the entire data distribution, which often leads to reliance on globally shared features while neglecting local patterns that may be critical for certain subsets of the data. In contrast, personalized model captures local patterns effectively, resulting in improved performance.
an upper bound for the federated learning (FI.) setup, particularly in the
Table Average test accuracy PMs and GM for clients Federated Learning (FL) setup and test accuracy non-FL Centralized Learning setup on FMNIST, CIFAR10, 10, and CIFAR100 datasets, Test accuracies are presented in percentage. NN refers to determinstic neural networks.
FMNIST
CIFARIO
CIFAR100
Method
PM (%)
GM (%)
PM (%)
GM (%)
PM (%)
GM (%)
FL (Ours; 50 Clients)
92.2 0.3
86. 0.4
77.1 =
65.9 0.9
66.6 0.4
56.0 +
Non-FL FL (Centralized; NN)
88.6 = 0.2
68.3 + 0.4
57.3 = 0.5
18

---

Harnessing Heterogencous Statistical Strength for Personalized Federated Learning via Hierarchical Bayesian Inference
Algorithm I Our Method
Input:
K: Total number of clients indexed by k
T: Total number of communication rounds
C: Client sampling rate
Eg: Total number of local base epochs
Ex: Total number of local head epochs
Eg Total number of global epochs
Initialize:
Global BNN model characterized by Global posterior distribution parameters 0"
Local BNN model characterized by Personalized posterior distribution for all client (0)) Aa E
for i 7 1 do
Server executes:
for k to K in parallel do
+ + PiientUpdate(k 0')
g)
Select random subset of clients using Binomial distribution B(K, C)
0
ServerUpdate (0 K' "x'")ee ess))
ServerUpdate( ( !k'es:):
& Optimize global BNN model's base component distribution parameters using Equation (9)
for 1 to Eg do
0 r+1
argmin L.g
(, giinner Ø g:head: 10 h Es:)
gibass
g:bave
Optimize global BNN model's 's head component distribution parameters using Equation (9) for to Eg do
1+1
argmin (6 (0 gibaser Φ² g:head (0-+1) TR es:)
gcheed
@gihead
1-1 g+1 x+1
gibase g:head
return 9
ClientUpdate(R: 0
Initialize local BNN model's base component distribution parameters
0' g:baae
kabbae
Optimize local BNN model's head component distribution parameters using Equation (10)
for = 1 to E,, do
0++1
kchoad
arrgmin (+
k:baser O Oshhead: O'
g:head)
Olcbond
Optimize local BNN model's base component distribution parameters using Equation (10)
for = to E, do
argmin La (O'ssbase
1++ Aiheadd ① bosse)
k.base
0 g++
1 Axbase k:head
return a 1
19


Reviews:
[{"review_id": "mTHNAeTwg0", "reviewer": "Reviewer_7w15", "summary": "The paper introduces a hierarchical Bayesian inference framework for personalized federated learning (pFL) that specifies personalized posterior parameters, enabling joint computation of global and local posteriors to balance personalization and global robustness. While theoretically subsuming existing Bayesian pFL methods (e.g., `pFedBayes`, `pFedVEM`), it can produce superior global model (GM) and personalized model (PM) performances under statistical heterogeneity.\n\n## Update after rebuttal\n---\nI updated the score to **accept** because the authors made a great effort to address all of my concerns.\nI have summarized below how my evaluation changes after the rebuttal:\n* The authors have faithfully performed additional experiments and theoretical analyses according to the review.\n* Thus, I can happily conclude that both empirical and theoretical improvements are made during the rebuttal period.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 7W15 for taking the time to provide a constructive and detailed review.\n\n**\"..only a simple separation has been provided..\"**: For fair comparison, the base-head setting in our framework follows (Collins et al., 2021) and pFedVEM (Zhu et al., 2023). As Reviewer suggested, we investigate the impact of the updating orders of the PM's base and head. The performance on CIFAR10 with 50, 100, and 200 clients are as below:\n|Updating Order|Sharing Component|50 Clients (PM)|50 Clients (GM)|100 Clients (PM)|100 Clients (GM)|200 Clients (PM)|200 Clients (GM)\n|--|--|--|--|--|--|--|-- \n|Base Only (Oh et al., 2022)|Base Only|76.8±0.2|-|72.9±0.2|-| 68.3±0.4|-\n|Base->Head|Both Head & Base|76.7±0.4|65.1±0.5|72.6±0.8|60.6±0.6|68.3±0.4|57.3±0.6 \n|Head->Base|Both Head & Base |**77.1±0.1**|**65.9±0.9**|**74.7±0.3**|**63.8±0.3**|**70.4±0.2**|**59.5±0.7**\n\nThe results show that the setting in our paper, as in the third row, is the best one. Note that if we only share base or head, the global model will be incomplete. \n\n**\"..how such a subsumption helps or improves existing methods..\"**: The benefits from our general framework are as follows:\n- we proved that one of the constraints of the existing methods is that they implicitly set the regularization coefficients $\\lambda_1$ and $\\lambda_2$ to $0$. As in Table 2, our ablation study shows that $\\lambda_1=\\lambda_2=0$ leads to inferior performance. \n- Through the principled general framework we proposed, it shed light on the relationships and limitations of the existing methods. \n- we derived the relationship between the coefficients and the hyper-parameters, showing how they govern the statistical strength of the sharing process. \n\n**\"..the strength of Bayesian inference..uncertainty quantification\"**: Following (Hahn, Jeong and Lee, 2022) setting, we quantify prediction uncertainty under symmetric label flipping with a noise ratio of 0.2 on cifar10: \n|Methods|50 Clients: ECE (Acc)|100 Clients: ECE (Acc)\n|--|--|--\n|FedAvg (GM)|0.37±0.06 (36.8±0.9)|0.35±0.08 (37.9±2.5)\n|pFedVEM (PM)|**0.13±0.01** (70.1±0.4)|0.17±0.0 (68.0±0.4)\n|Ours (PM)| 0.15±0.0 (**72.6±0.4**)|**0.16±0.01 (70.3±0.3)**\n\nOur method's uncertainty quantification is comparable with pFedVEM but achieves higher accuracy.\n\n**\"Theoretical convergence analysis is absent..\"**: Evaluating the statistical convergence of the local personalized posteriors, we can show that the posterior mean $E[\\theta_k|D]$ is in between the MLE estimate $\\hat{\\theta}_k $ and the global mean $\\mu_g$: $E[\\theta_k|D_k,\\mu_g,\\tau^2_g]=\\alpha_k\\mu_g+(1-\\alpha_k)\\hat{\\theta}_k$ where the convergence speed towards the global mean is given by $\\alpha_k=\\frac{\\sigma^2_k}{\\sigma^2_k+\\tau^2_g}$. It shows there is more shrinkage for clients with smaller measurement precision. \n\n**\"..justification of..half-normal distribution\"**: half-normal is commonly used as a prior over the standard deviation of Gaussian. As a special case of inverse-Wishard dist., conjugate over Gaussian variance, half-normal is computationally more stable and more efficient.\n\n**\"The approximation of $q(\\theta_g|\\phi)$..\"**: Simplifying $q(\\theta_g|\\phi)$ to a delta function improves communication efficiency without sacrificing performance. Using the delta function, we only need to broadcast a single $\\theta_g$ value back to the clients. With the original variational distribution, it needs to broadcast a set of samples to each client.\n\n**\"..requires to tune the hyperparameters..**: Although tuning $\\lambda_1$ and $\\lambda_2$ incurs additional computation overhead as discussed in Appendix A.4, the performance gain is statistically significant across benchmarks. \n\n**\"..validation with a moderate number of clients..\"**: We validated our framework with the largest number of clients (i.e., 200) among the baseline methods on the image datasets (e.g., pFedBayes with maximum 20 clients, FedAvg 100 clients, and pFedVEM 200 clients). As Reviewer suggested, we conduct experiments with 1000 clients on Cifar10 as below:\n|Methods| PM | GM\n|--|--|--|\n|FedAvg|-|48.7±1.1\n|pFedVEM|59.0 ± 0.4|48.6±0.9\n|Ours|**61.0 ± 0.3**| **52.3±0.4**\n\nas shown above, our method scales well with large client size.\n\n**\"why the PM performance does not outperform other methods..in Figure 2\"**: The synthetic datasets are relatively simple in terms of lower feature dimensionality and smaller dataset sizes. All the baselines can perform well in terms of capturing the non-IID information.\n\n**\"local computation overhead..for local BNN training\"**: Average local training times (seconds) of BNNs per communication round on Cifar10 with all 100 clients available are: \n|Methods|Time\n|--|--\n|non-Bayes (FedAvg)|21.42\n|non-Bayes Base, Bayes Head (pFedVEM)|43.14\n|full-Bayes (Ours, pFedBayes)|90.38\n\n**\"comparison of the communication cost..\":** the communication cost comparison was reported in Figure 3 and in Section 6.2.1.\n\nWe will clarify the above points and correct typos and notations in the camera-ready version.", "claims_and_evidence": "### Main Claim\n---\nThe proposed hierarchical Bayesian framework i) outperforms existing pFL methods in terms of both GM and PM performances by ii) effectively sharing a model weights while retaining local-specific knowledge, and iii) also subsumes existing Bayesian pFL methods.\n\n### Breakdown\n---\ni) pFL Performance\n* This main claim is well supported by empirical performances in Table 1 (Page 6) and Figure 2 (Page 5) even under practical setup, involving heterogeneity adjusted by $(\\alpha, \\beta)$ and partial client participation.\n* In Figure 4, it shows better PM accuracy across varying local data sizes on benchmark datasets, indicating better collaboration benefits of the proposed method.\n\nii) Effective Sharing Strategy\n* The authors adopted base-head separation, which is widely used in practice, for the effectiveness of pFL.\n* However, only a simple separation has been provided, and none of the ablation (e.g. the order of updating, only sharing either the head or the base) or in-depth analysis/justification is provided.\n\niii) Subsumption\n* In Section 4, the authors neatly expanded that their proposed method generalizes existing Bayesian pFL methods.\n* Describing how such a subsumption helps or improves existing methods is more helpful in highlighting the proposed method, but such expositions are lacking in the current form.", "methods_and_evaluation_criteria": "Yes, the proposed method and the evaluation criteria follow a typical scheme of pFL methods, and the authors properly simulated statistical heterogeneity due to label skewness and concept drift, both of which well reflect the real-world situation.", "theoretical_claims": "Theoretical convergence analysis is absent, and only relies on empirical convergence results (Figure 3, 4 and 6).", "experimental_designs_or_analyses": "Although all experiments are limited to image classification benchmark datasets with a moderate number of clients (i.e. <200 clients), this is a common setup in the FL literature and is therefore acceptable.", "relation_to_broader_scientific_literature": "FL typically assumes a point estimate of a global model parameter. With the contribution of the paper, FL can be extended to find the posterior distribution of a global model parameter, which can contribute to Bayesian inference-related tasks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n* The proposed method subsumes existing Bayesian pFL methods.\n* The variational inference-based optimization is easy to be implemented in practice thanks to Algorithm 1.\n\nWeaknesses\n* There is lack of justification of why the hyper prior distribution should be half-normal distribution.\n* The approximation of $q(\\boldsymbol{\\theta}_g;\\phi)$ simplified Eq. 8, however, it can possibly affect to the overestimate of GM weights, high sensitivity to initialization, etc. Thus, proper correction method could be exploited, e.g., Laplace approximation, temperature scaling.\n* The proposed method requires to tune the hyperparameter of hyper prior distribution, i.e., $\\lambda_1, \\lambda_2$.\n* The proposed method has only been validated with a moderate number of clients, which may not fully reflect a large-scale cross-device FL setup.\n* The proposed method only shows the pFL performance, and bypassed the strength of Bayesian inference (e.g., uncertainty quantification).", "other_comments_or_suggestions": "Please see 'Questions For Authors' section.", "questions_for_authors": "* In what practical situation can we consider using the proposed method? The Bayesian perspective of pFL is intriguing in itself, and I believe that a complementary practical impact would also increase the validity of the proposed method. For example, authors could consider adding uncertainty quantification-related benchmark experiments under noisy-label scenario (e.g., see experimental setup of 'robustness to label noise' section in (Hahn, Jeong and Lee, 2022))\n* Could the authors please provide any reasons of why the PM performance does not outperform other methods like PM did in Figure 2?\n* How much local computation overhead is required for local BNN training, i.e., the computation of Eq. (10), compared to the non-Bayesian setup, e.g., `FedAvg`?\n* Please also consider adding a comparison of the communication cost/complexity of the proposed method to that of `pFedBayes` and `pFedVEM`.\n* I presume that the order in which the base and head are updated is important for pFL performance. In Algorithm 1, lines 807-823, the authors suggest updating the head first and the base later. Could you please provide simple comparative results when i) the base is updated first and the head later, when ii) only the base is updated and communicated, following (Collins et al., 2021) and (Oh et al., 2022)? If approach ii) shows decent performance, I think it can contribute to a lightweight Bayesian posterior update of the current proposed method.\n* Is there any justification for choosing the half-normal distribution as the prior rather than the (inverse) Wishart distribution? Any supporting evidence (e.g. appropriateness for hierarchical Bayesian inference, empirical comparisons, etc.) would be appreciated.\n* In the caption of Figure 1, 'Overview of our PFL framework based on hierarchical Bayesian inference (right)' -> 'Overview of our PFL framework based on hierarchical Bayesian inference (**left**)'\n* Please avoid using $(\\alpha,\\beta)$ in adjusting the statistical heterogeneity as it overlaps with the notation of hyperparameter of a hyper prior distribution defined in Eq. 3.\n\n\n> (Hahn, Jeong and Lee, 2022) Connecting Low-Loss Subspace for Personalized Federated Learning  \n> (Collins et al., 2021) Exploiting shared representations for personalized federated learning  \n> (Oh et al., 2022) FedBABU: Towards Enhanced Representation for Federated Image Classification", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kj0pMTCIvi", "reviewer": "Reviewer_DMYx", "summary": "The paper proposes a hierarchical Bayesian inference framework for personalized federated learning (PFL), addressing the issue of statistical heterogeneity across decentralized client datasets. The main conceptual contribution is introducing a conjugate hyperprior over personalized posterior parameters, allowing simultaneous inference of local personalized models and global aggregated models. This approach generalizes prior Bayesian methods for federated learning, aiming to balance local model personalization with global model robustness. Experimental results show competitive performance, especially in global model accuracy.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer DMYx for taking the time to provide a review on our paper.\n\n**\"...personalized models show only modest or comparable improvements\":** The results in Table 1 (Page 6) shows the personalized models (PMs) of our framework achieves significant better performance than the baseline methods across all the benchmark datasets. Our PM's performance is comparable with the baselines only on the synthetic data. This is because the synthetic data are relatively simple in terms of lower feature dimensionality and smaller dataset sizes than the benchmark data. All the baselines can perform well in terms of capturing the personalized information.\n\n**\"Bayesian methods inherently incur high computational costs...:** This general claim about Bayesian inference is not necessarily applicable for FL problems. Figure 3 (Page 7) shows that our method achieves better performance with significant fewer communication rounds than other Bayes and non-Bayes FL approaches. The average local training time (seconds) per communication round in Cifar10 with all 100 clients available is\n|Methods|Time\n|--|--\n|non-Bayes (FedAvg)|21.42\n|non-Bayes Base, Bayes Head (pFedVEM)|43.14\n|full-Bayes (Ours,pFedBayes)|90.38\n\nSo the overall computational cost is not necessarily higher than non-Bayes FL methods. \n\n**\"I find the novelty of the proposed approach limited...\":** The novel contributions of our work are \n- we propose a novel hierarchical Bayesian inference framework for personalized federated learning (pFL), introducing a hyper-prior over personalized posterior parameters;\n- we develop a stochastic variational inference scheme for the framework, enabling joint computation of global and local posteriors to balance local personalization and global robustness; \n- we subsume the existing Bayesian pFL methods as special cases of our general framework, shedding light on their relationships and limitations;\n- our framework significantly outperforms existing pFL methods in terms of both GMs and PMs across the benchmark datasets. \n\nWe will clarify the novel contributions in the camera-ready version.\n\n\n**\"the selected baselines do not explicitly require personalization, and better results could be achieved by training a single model with access to all data, suggesting that the primary challenge with the chosen datasets may lie in optimization rather than modeling methodology\":** \n- The benchmark datasets we used are widely adopted for evaluating FL methods. Furthermore, as in Section 6.2, we applied common data partitioning approaches to examine various categories of statistical heterogeneity that exist in real-world FL scenarios including label distribution skew (i.e., clients possess varying label distributions), label concept drift (i.e., feature distributions differ among clients), and data quantity disparity (clients have different amounts of data). This is obviously a more challenging problem scenario than \"training a single model with access to all data\".\n- Figure 4 (Page 8) shows that our method provides greater benefits for the clients to participate in the collaborative training process in comparison with training a single model on its own or the competing FL approaches. \n- Although our work focuses on personalized FL, we included both personalized and non-personalized state-of-the-art FL methods as baselines for performance comparison.\n\n**\"Bayesian methods generally demand significantly greater computational resources and present scalability issues...\":** To the best of our knowledge, there is no evidence suggesting Bayesian FL suffers from \"greater computational resources and present scalability issues\". We validated and tested our framework with the largest number of clients (i.e., 200) among the FL literature reported on the image datasets. To further show the scalability of our method, we conduct experiments with a total of 1000 clients using the Cifar10 dataset, where each client has unique local data of 5 labels out of 10., and report the results below: \n\n|Methods| PM | GM\n|--|--|--|\n|FedAvg | -| 48.7 &plusmn; 1.1\n|pFedVEM| 59.0 &plusmn; 0.4|48.6 &plusmn; 0.9\n|Ours| **61.0 &plusmn; 0.3**| **52.3 &plusmn; 0.4**\n\nAs shown in the table our method scales well with even 1000 clients and still outperfroms both Bayes FL (pFedVEM) and non-Bayes FL (FedAvg). \n\n**\"non-Bayesian approaches, which can leverage much larger network sizes due to their efficiency, should also be included in the evaluations to provide a fairer comparison.\"** In our performance comparison in Table 1 and communication efficiency comparison in Figure 3, we reported state-of-the-art performance of both Bayes and non-Bayes FL methods. The non-Bayes FLs include FedAvg, pFedMe, FedRep, etc. The results shows that our method achieves significant better performance with less communication rounds. Using larger network sizes for non-Bayes FL will also incurs higher computational cost and not necessarily achieves better performance for non-IID FL data settings.", "claims_and_evidence": "The paper's primary claim—that the hierarchical Bayesian approach effectively harnesses statistical heterogeneity—is supported through extensive empirical evaluation. However, the claim of significantly improved personalized model performance compared to existing methods is not convincingly demonstrated. While the global model does perform robustly, personalized models show only modest or comparable improvements.", "methods_and_evaluation_criteria": "The experimental design is generally sound. Authors evaluated the method across synthetic and real datasets (Fashion-MNIST, CIFAR-10, CIFAR-100), assessing statistical heterogeneity and scalability with varied client numbers. However, a significant limitation is that Bayesian methods inherently incur high computational costs, which was not explicitly compared against simpler, non-Bayesian baselines using larger networks or alternative optimization methods.", "theoretical_claims": "I checked the theoretical derivations of the hierarchical Bayesian inference framework and the special-case reductions to existing methods (pFedBayes and pFedVEM). The derivations appear correct, clearly showing how previous methods can be recovered as special instances of the proposed model under specific constraints. No major theoretical issues were observed.", "experimental_designs_or_analyses": "Considering the existing body of work on Bayesian inference for federated learning, I find the novelty of the proposed approach limited, and the empirical results presented are not particularly strong. Specifically, the method does not significantly improve personalized model accuracy—the most crucial metric in personalized federated learning—compared to existing approaches. Performance improvements, if any, are minimal. Furthermore, the selected baselines do not explicitly require personalization, and better results could be achieved by training a single model with access to all data, suggesting that the primary challenge with the chosen datasets may lie in optimization rather than modeling methodology. Additionally, Bayesian methods generally demand significantly greater computational resources and present scalability issues. Consequently, non-Bayesian approaches, which can leverage much larger network sizes due to their efficiency, should also be included in the evaluations to provide a fairer comparison.", "relation_to_broader_scientific_literature": "The main conceptual contribution is introducing a conjugate hyperprior over personalized posterior parameters, allowing simultaneous inference of local personalized models and global aggregated models. This approach generalizes prior Bayesian methods for federated learning, aiming to balance local model personalization with global model robustness. Experimental results show competitive performance, especially in global model accuracy.", "essential_references_not_discussed": "To the best of my knowledge, the related work is well covered.", "other_strengths_and_weaknesses": "See my review above.", "other_comments_or_suggestions": "See my review above.", "questions_for_authors": "See my review above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XKthzGWmxP", "reviewer": "Reviewer_5j3i", "summary": "This paper proposes a novel hierarchical Bayesian inference framework for Personalized Federated Learning (PFL) by specifying a conjugate hyper-prior over the parameters of personalized posteriors. This approach enables the joint computation of a global posterior distribution for aggregation, alongside personalized posteriors at the local level. The proposed framework strikes an elegant balance between local personalization and global model robustness, offering a more efficient and scalable solution for PFL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 5j3i for taking the time to provide detailed and constructive review on our paper.\n\n**\"the baseline portion of the experiment should be compared more with the latest approaches to personalized federated learning\"**: To the best of our knowledge, we included the latest competing Bayesian and non-Bayesian FL methods as the baselines. In addition, we show the performance comparison below with two pFL published in 2024. Please note that they are less comparable because both of them require additional datasets to pre-train the GMs.  \n|Method|Number of clients|Cifar10 PM|Cifar100 PM\n|--|--|--|--|\n|FedBNN [1] |20|**79.9 &plusmn; 0.3**|57.3 &plusmn; 0.8\n|FedPPD [2] |10|61.86|53\n|Ours|50|77.1 &plusmn; 0.1| **66.6 &plusmn; 0.4**\n\nBy aggregating information from even more clients (more clients, more challenging the FL task is) we still achieve comparable or better performance in Cifar100.\n\n**\"evaluate the model’s performance on datasets with a larger number of categories, such as Tiny-ImageNet,..\"**: As suggested by the reviewer, we conduct experiments on the Tiny-ImageNet dataset with 50 clients, where each client has unique local data consisting of 50 labels out of 200, and report the results below: \n\n|Methods| PM | GM\n|--|--|--|\n|FedAvg | -| 10.1 &plusmn; 0.7\n|pFedVEM|33.45 &plusmn; 0.2|18.5 &plusmn; 0.4\n|Ours| **42.4 &plusmn; 0.1**| **25.7 &plusmn; 0.1**\n\nIt shows both PM and GM of our framework achieve the best performance. We will add the results in the camera-ready version.\n\n**\"in Table 1, the global model performs poorly on FMNIST when the number of clients is 100 and 200..\"** Our global model is the second best on FMNIST. This is mainly because the FMNIST dataset is relatively simple and does not have many non-IID features. Therefore, the advantages of personalization algorithms are not obvious. This can be verified by the fact that all the baselines including FedAvg achieves similar performance. We will clarify this in the camera-ready version.\n\n**Abbrevations and typos:** We will provide explanations for the abbrevations, and correct all the typos in the camera-ready version. \n\n**\"does the method assume independence between global and local parameters?\"**: As discussed  in Section 3.2 Line 126-128, the local parameters of client k’s personalized posterior $\\theta_k$ are direct instantiations of the hyper-prior $p(\\theta_g|\\alpha)$. So they are dependent on the global parameters.\n\n**\"Hierarchical Bayesian inference requires computing the global posterior distribution on the server and frequently synchronizing client parameters\"**: As discussed in Section 3.3, we split the overall ELBO into server part and client part to avoid the potential communication overhead caused by frequently synchronizing client parameters. Because the client part of the ELBO can be optimized locally in multiple rounds after receiving updates from the server.\n\nWe will clarify the points in the camera-ready version.\n\n**References**\n\n[1] Makhija, Disha, Joydeep Ghosh, and Nhat Ho. \"A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings.\" Advances in Neural Information Processing Systems 37 (2024): 102428-102455.\n\n[2] Bhatt, Shrey, Aishwarya Gupta, and Piyush Rai. \"Federated learning with uncertainty via distilled predictive distributions.\" Asian Conference on Machine Learning. PMLR, 2024.", "claims_and_evidence": "This paper proposes a novel hierarchical Bayesian inference framework for Personalized Federated learning. I believe that the claims made in the submission are well supported by evidence, but need further refinement (see comments)\n\nComments\n\n1.Because the authors want to improve local personalized learning while ensuring the ability to generalize the global model, the baseline portion of the experiment should be compared more with the latest approaches to personalized federated learning.\n\n2.The experiments in this paper are primarily conducted on the FMNIST, CIFAR-10, and CIFAR-100 datasets. It is recommended to evaluate the model’s performance on datasets with a larger number of categories, such as Tiny-ImageNet, to assess its generalizability. Additionally, in Table 1, the global model performs poorly on FMNIST when the number of clients is 100 and 200. It would be necessary for the authors to provide an explanation for this phenomenon.\n\n3.The paper frequently uses the abbreviation 'BNN' without specifying its full form. For clarity and consistency, it would be helpful to define 'BNN' explicitly when it is first introduced.\n\n4. A minor typo issue in the caption of Figure 1. \"Our PFL framework is on the left, not the right.\"", "methods_and_evaluation_criteria": "The method proposed in this paper preserves the client's personalized features and ensures the global generalization capability, but it needs further improvement.", "theoretical_claims": "No separate theoretical analysis section in the text.", "experimental_designs_or_analyses": "1.Because the authors want to improve local personalized learning while ensuring the ability to generalize the global model, the baseline portion of the experiment should be compared more with the latest approaches to personalized federated learning.\n\n2.The experiments in this paper are primarily conducted on the FMNIST, CIFAR-10, and CIFAR-100 datasets. It is recommended to evaluate the model’s performance on datasets with a larger number of categories, such as Tiny-ImageNet, to assess its generalizability. Additionally, in Table 1, the global model performs poorly on FMNIST when the number of clients is 100 and 200. It would be necessary for the authors to provide an explanation for this phenomenon.", "relation_to_broader_scientific_literature": "This paper proposes a personalized federated learning method based on hierarchical Bayesian inference, which is of significance in helping the model adaptively adjust the trade-off between sharing and personalization among different clients by constructing a shared global prior and a personalized posterior.", "essential_references_not_discussed": "The paper comprehensively discusses the prior work related to its main contributions, appropriately cites and compares relevant approaches in the area of bayesian federated learning and personalized federated learning, and does not omit any important references that will have a significant impact on the understanding of the proposed approach.", "other_strengths_and_weaknesses": "Strengths: \n\n1. By defining the conjugate super-prior on the distribution of personalized a posteriori parameters, the article achieves global model optimization while preserving the personalized features, and verifies the effectiveness of the method on three datasets.\n\n2. The article demonstrates that the existing Bayesian PFL methods are special cases of its framework, and the optimization objectives of these methods can be obtained by introducing additional constraints, which provides a unified perspective for subsequent research.\n\nWeaknesses:\n\n1.In Equation (5), for the sake of computational simplification, does the method assume independence between global and local parameters? In practical scenarios, there is likely some degree of dependency between them.\n\n2.Hierarchical Bayesian inference requires computing the global posterior distribution on the server and frequently synchronizing client parameters. When the number of clients K is large, this may introduce significant communication overhead.", "other_comments_or_suggestions": "I noticed a minor typo issue in the caption of Figure 1. Our PFL framework is on the left, not the right.", "questions_for_authors": "I'll reorganize all my queries here\n1.Because the authors want to improve local personalized learning while ensuring the ability to generalize the global model, the baseline portion of the experiment should be compared more with the latest approaches to personalized federated learning.\n\n2.The experiments in this paper are primarily conducted on the FMNIST, CIFAR-10, and CIFAR-100 datasets. It is recommended to evaluate the model’s performance on datasets with a larger number of categories, such as Tiny-ImageNet, to assess its generalizability. Additionally, in Table 1, the global model performs poorly on FMNIST when the number of clients is 100 and 200. It would be necessary for the authors to provide an explanation for this phenomenon.\n\n3.The paper frequently uses the abbreviation 'BNN' without specifying its full form. For clarity and consistency, it would be helpful to define 'BNN' explicitly when it is first introduced.\n\n4.In Equation (5), for the sake of computational simplification, does the method assume independence between global and local parameters? In practical scenarios, there is likely some degree of dependency between them. It would be beneficial to clarify whether this assumption affects the model's expressiveness.\n\n5.Hierarchical Bayesian inference requires computing the global posterior distribution on the server and frequently synchronizing client parameters. When the number of clients K is large, this may introduce significant communication overhead.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sY5G4p3ga1", "reviewer": "Reviewer_kx6d", "summary": "This paper proposes a hierarchical Bayesian inference framework for personalized federated learning (PFL) to address issues arising from statistical heterogeneity across client data. The main contribution involves specifying a conjugate hyper-prior over personalized posterior parameters, enabling joint inference of personalized and global posterior distributions. Experimental results across synthetic and real datasets (FMNIST, CIFAR10, CIFAR100).  The authors further demonstrate analyctically that existing Bayesian PFL methods (e.g., *pFedBayes* and *pFedVEM*).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer kx6d for taking the time to provide detailed and constructive review on our paper.\n\n**\"..it is not entirely clear why the addition of global hyperpriors significantly outperforms simpler aggregation methods \"**: We show that simpler aggregation methods implicitly set the regularization coefficients $\\lambda_1$ and $\\lambda_2$ to $0$. As in Table 2 (Page 8), our ablation study shows that $\\lambda_1=\\lambda_2=0$ leads to inferior performance. Simpler aggregation averages out the local posterior parameters. This hinders their ability of capturing non-IID features since the arithmetic means blur the statistical heterogeneity of the local data. Defining a hyperprior over the personalized parameters lets us share their statistical strength through the hyperprior and retaining the unique properties of local models. \n\n**\"...placing a hierarchical prior...introduces significant computational overhead compared to simpler methods like pFedBayes...its justification are not adequately discussed.\"**: As described in Section 3.3, we split the overall ELBO into server part and client part to avoid the potential communication overhead caused by frequently synchronizing client parameters, since the client part of the ELBO can be optimized locally in multiple rounds after receiving updates from the server. Comparing with simpler methods, the only additional computation overhead, as discussed in Appendix A.4, is about tuning the regularization coefficients $\\lambda_1$ and $\\lambda_2$ via grid search. This can be justified by the statistically significant performance gain across benchmarks. \n\n**\"Novelty might be perceived as limited...in terms of these two papers\"**: Although paper (Kim, et al. 2023) proposes to define Normal-inverse Wishard (NIW) distribution and mixture models as hyper-priors over personalized posterior parameters, in the implementation, they simplify the NIW to \"spiky Gaussian\" by fixing the covariance to a singular identity matrix, as in their Equation (16). This means they fail to aggregate the covariances of the personalized posteriors. This is also verified by the poorer performance the paper reported in their experiments. In comparison, we propose half-Gaussian, a special case of NIW, over the covariances and effectively aggregate the personalized posteriors. Besides our more comprehensive experiments, we also provide theoretical analysis shedding light on the relationships and limitations of the existing Bayesian pFL methods. Paper (Hassan, et al. 2023) introduces a FL model with hierarchical latent variables, instead of directly performing Bayesian inference on distribution parameters as in our paper. The work's empirical evaluation is quite limited, since the model is not tested on any benchmark datasets. \n\nAlthough the two papers are arXiv drafts, we will discuss them in our camera-ready version and clarify the novelty of our work. \n\n**\"Computational complexity and scalability aspects are not explicitly addressed\"**:\n- Regarding the computational complexity, Figure 3 (Page 7) shows our method outperforms the baselines with fewer communication rounds. The average local training time (seconds) per communication round in Cifar10 with all 100 clients available is\n|Methods|Time\n|--|--\n|non-Bayes (FedAvg)|21.42\n|non-Bayes Base, Bayes Head (pFedVEM)|43.14\n|full-Bayes (Ours, pFedBayes)|90.38\n\n- For scalability, we validated and tested our framework with the largest number of clients (i.e., 200) in comparison with what the baseline methods reported on the image datasets. For example, pFedBayes uses maximum 20 clients, FedAvg 100 clients, and pFedVEM 200 clients. To demonstrate the scalability, we conduct an experiment with a total of 1000 clients on Cifar10, where each client has unique local data of 5 labels out of 10, as below:\n|Methods| PM | GM\n|--|--|--|\n|FedAvg | -| 48.7±1.1\n|pFedVEM| 59.0±0.4|48.6±0.9\n|Ours| **61.0± 0.3**| **52.3±0.4**\n\nOur method scales well with large number of clients.\n\n**\"Are there specific scenarios or practical constraints where the proposed hierarchical Bayesian approach might underperform compared to simpler methods?\":** Our PMs' performance is comparable with the baselines on the synthetic data. This is because the synthetic data are relatively simple in terms of lower feature dimensionality and smaller dataset sizes. The baselines can thus perform well capturing the non-IID information. The similar case is on FMNIST dataset which is relatively simple and does not have many non-IID features. Therefore, the advantages of personalization algorithms are not obvious. This can be verified by the fact that all the baselines including FedAvg achieves similar performance.\n\n**\"Could you clarify how much performance improvement is directly attributable to the hierarchical hyperpriors versus the optimization method itself?\"**: Our ablation study in Table 2 (Page 8) suggests that properly setting the parameters of the hyperprior is important to the performance gain.", "claims_and_evidence": "The claims of achieving superior performance and addressing statistical heterogeneity are well-supported by empirical evaluations on synthetic and standard benchmark datasets (FMNIST, CIFAR10, CIFAR100). However, it is not entirely clear why the addition of global hyperpriors significantly outperforms simpler aggregation methods such as that from pFedBayes. Since the aggregation of existing methods correspond to particular priors under certain assumptions, it is unclear how an uninformative hyperprior would yield substantial performance improvements.", "methods_and_evaluation_criteria": "The methods and the evaluation criteria are appropriate for assessing performance in personalized federated learning scenarios. The use of benchmark datasets (FMNIST, CIFAR10, CIFAR100) and synthetic data for controlled heterogeneity simulations aligns well with the targeted problems.", "theoretical_claims": "I checked the claim regarding the special cases (the derivation showing that pFedBayes and pFedVEM methods are a special case of the proposed hierarchical Bayesian formulation).", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound. The use of different client counts (50, 100, and 200) and varying data distributions to simulate realistic federated settings was well-justified. No significant issues were found in the experimental setups.", "relation_to_broader_scientific_literature": "The paper clearly situates itself within the broader scientific literature by discussing connections and distinctions to related Bayesian-based FL methods and personalized FL approaches. However, essentially this method appears primarily to introduce a prior over local parameters, a concept already capable with existing algorithms and methodlogies in the literature, though maybe not quite *explored* in directly the same manner (see references below).  Additionally, while placing a hierarchical prior might be conceptually principled, it introduces significant computational overhead compared to simpler methods like pFedBayes, which typically avoid full stochastic VI both locally and globally within each communication round. This additional computational cost and its justification are not adequately discussed.", "essential_references_not_discussed": "I think these two papers can do the model (or model type) that this paper describes. And I think the authors should really try and readdress their novelty in terms of these two papers. At the very least, these papers should be referenced and discussed within: \n\n- Kim, Minyoung, and Timothy Hospedales. \"Fedhb: Hierarchical bayesian federated learning.\" arXiv preprint arXiv:2305.04979 (2023).\n- Hassan, Conor, Robert Salomone, and Kerrie Mengersen. \"Federated variational inference methods for structured latent variable models.\" arXiv preprint arXiv:2302.03314 (2023).", "other_strengths_and_weaknesses": "**Strengths:**\n- Hierarchical Bayesian approaches are elegant and address statistical heterogeneity effectively.\n- Comprehensive empirical studies clearly demonstrate the benefits and effectiveness of the method.\n- The method is well-motivated, and the connections to existing methods are clearly explained.\n\n**Weaknesses:**\n\n- Novelty might be perceived as limited since many works propose models and algorithms for federated learning of hierarchical models nowadays. \n- Computational complexity and scalability aspects are not explicitly addressed.\n- The additional computational cost of optimizing global hyperpriors is not clearly justified, particularly compared to simpler arithmetic aggregation methods (e.g., pFedBayes). \n- Without highly informative priors, it is unclear how substantial performance improvements are achieved.", "other_comments_or_suggestions": "**Other comments or suggestions:** Clarifying the computational complexity of the algorithm, compared to existing methods, would strengthen the work.", "questions_for_authors": "- Could you elaborate on the computational complexity of the hierarchical Bayesian method compared to existing methods?\n- Could you elaborate on the novelties of your model and algorithm formulation relative to the two references that I included?\n- Are there specific scenarios or practical constraints where the proposed hierarchical Bayesian approach might underperform compared to simpler methods?\n- Could you clarify how much performance improvement is directly attributable to the hierarchical hyperpriors versus the optimization method itself?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a Hierarchical Bayesian approach to personalized federated learning, with hyperpriors over the local Gaussian Bayesian neural networks. 

Reviewers diverge in their view of this work. Reviewer 7w15 increased their score from 2 to 4 during the rebuttal, as they felt like the authors addressed their concerns such as computation cost and additional benefits of being Bayesian. 

Reviewer DMYx recommends reject, but in my opinion, the authors have rebutted the reviewer's concerns on computation cost of the method, and the point about personalized vs global models is not so important. The authors can consider adding a global baseline (as if all data was available in one place), which would be an upper bound on global performance, and which I think all personalized FL papers should have anyway (to see what room there is for benefits of having personalized models). They (and other reviewers) point out the additional cost of tuning two new hyperparameters but I think that this is a limitation of the method that I commend the authors for explicitly mentioning and including an ablation. 

Reviewers 5j3i and kx6d stuck with their scores after rebuttal but I think the authors have mostly rebutted their points. The paper will be stronger after including the additional results and discussion as promised by the authors, including comparison to 2 works as mentioned by reviewer kx6d, computation time results, and additional results with larger number of clients and larger benchmark. 

Overall, I think the authors can more clearly give intuition as to why the existing methods that are part of their framework are worse than their method. They provide some justification in the rebuttal about lambda1=lambda2=0 but this was not sufficiently concise for me, and this can be improved for the camera-ready version.
Decision: Accept (poster)