ICML 2025

Harnessing Heterogeneous Statistical Strength for Personalized Federated
Learning via Hierarchical Bayesian Inference
Mahendra Singh Thapa 1 Rui Li 1
Abstract
Personalized federated learning (PFL) based on
Bayesian approach tackle the challenges from sta-
tistical heterogeneity of client data by comput-
ing a personalized posterior distribution over the
parameters of each client’s local model and con-
structing a global distribution by aggregating the
parameters of these personalized posteriors. How-
ever, the heuristic aggregation methods introduce
strong biases and result in global models with
poor generalization. We thus propose a novel hi-
erarchical Bayesian inference framework for PFL
by specifying a conjugate hyper-prior over the
parameters of the personalized posteriors. This
allows us to jointly compute a global posterior
distribution for aggregation and the personalized
ones at local level. This hierarchical Bayesian
inference framework achieves elegant balance be-
tween local personalization and global model ro-
bustness. Extensive empirical study shows that by
effectively sharing the heterogeneous statistical
strength across the local models while retaining
their distinctive characteristics, our framework
yields state-of-the-art performance. We also show
that existing Bayesian PFLs are special cases of
our framework.
1. Introduction
Federated learning (FL) emerges as a promising framework
that enables collaborative model training across decentral-
ized client devices while preserving data privacy (McMahan
et al., 2017). This approach demonstrates significant poten-
tial in various real-world applications, such as healthcare
(Yang et al., 2021; Dayan et al., 2021; Yang et al., 2024),
recommendation systems (Muhammad et al., 2020; Yuan
1College of Computing and Information Sciences, Rochester
Institute of Technology, Rochester, New York, USA. Correspon-
dence to: Rui Li <rxlics@rit.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning...