Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approach
Authors: Mahrokh Ghoddousi Boroujeni
Andreas Krause
Giancarlo Ferrari-Trecate
Venue: Accepted by TMLR ()

Abstract: Federated Learning (FL) aims to infer a shared model from private and decentralized data stored by multiple clients. Personalized FL (PFL) enhances the model’s fit for each client by adapting the global model to the clients. A significant level of personalization is required for highly heterogeneous clients but can be challenging to achieve, especially when clients’ datasets are small. We introduce PAC-PFL for PFL of probabilistic models. PAC-PFL infers a shared hyper-posterior and treats each client’s posterior inference as the personalization step. Unlike previous PFL algorithms, PAC-PFL does not regularize all personalized models towards a single shared model, thereby greatly enhancing its personalization flexibility. By establishing and minimizing a PAC-Bayesian generalization bound on the average true loss of clients, PAC-PFL effectively mitigates overfitting even in data-poor scenarios. Additionally, PAC-PFL provides generalization bounds for new clients joining later. PAC-PFL achieves accurate and well-calibrated predictions, as supported by our experiments.

Published in Transactions on Machine Learning Research (02/2025)
Personalized Federated Learning of
Probabilistic Models: A PAC- PAC-Bayesian Approach
Mahrokh G. Boroujeni
Institute of Mechanical Engineering EPFL, Switzerland
Andreas Krause
krausea@etha.ch
Department of Computer Science ETH Zürich, Switzerland
Giancarlo Ferrari-Trecate
Institute of Mechanical Engineering EPFL, Switzerland
Reviewed on OpenReview: https: http::///ppreeevee net/ forum? <d- MMCr
Abstract
Federated Learning (FL) aims to infer a shared model from private and decentralized data stored by multiple clients. Personalized FL (PFL) enhances the model's fit. for each client by adapting the global model the clients. A significant level of personalization is required for highly heterogencous clients but be challenging to achieve, especially when clients' datasets are small. To address this issue, we introduce the PAC-PFL framework for PFL of probabilistic models. PAC-PFI. infers shared hyper-posterior and treats each client's s posterior inference as the personalization step. Unlike previous PFL algorithms,, PAC-PFL does not regularize all personalized models towards a single shared model, thereby greatly enhancing its personalization flexibility. By establishing and minimizing a PAC-Bayesian generalization bound on the average true Joss of clients, PAC-PFL effectively mitigates over- fitting even in data-poor scenarios. Additionally, PAC-PFL provides generalization bounds for new clients joining later. PAC-PFL achieves accurate and well-calibrated predictions, as supported by our experiments
1 Introduction
Federated Learning (FL) enables collaborative learning across decentralized datasets stored on end-devices, known as clients, without requiring raw data to be shared (Konečný et al., 2015). The primary objective of FL is to train a global model that performs well across all clients. The training process orchestrated by a trusted server, which iteratively distributes the current model to a subset of clients, aggregates their locally computed updates, and refines the model for subsequent rounds. By restricting access to only the communicated updates rather than the raw data itself, FL enhances privacy and reduces communication overhead compared to traditional centralized approaches.
A key challenge in FL is the hoterogeneity of clients' datasots, which violates the i.i.d. assumption required for training a global model [Kairouz et al. 2021 Li et al.] 2020a).. This often leads to convergence difficulties or suboptimal performance (Li et al.J 2020b).. To address this issue, Personalized FL (PFL.) introduces a personalization step adapt the global model to the specific data of individual clients. This step is critical
in many real-world federated datasets, as they typically involve heterogeneous clients (Wen et al. 2022
The codebase our algorithm available http:///tteee.goolee.//c//////e/

---

Published in Transactions on Machine Learning Research (02/2025)
The growing impact of PFL has been highlighted in diverse applications, including image classification, regression, text analysis, and recommendation systems (Chen et. al. 2024).
Despite significant advancements, several critical issues remain underexplored. First, (cl) most PFL ap- proaches yield point estimates, limiting their ability to quantify epistemic uncertainty, which is essential in safety-critical applications (Guo et al., 2017 Achituve et al.] (021). Second, (c2) personalized models are often closely tied to the global model, making them less effective in highly heterogeneous or multimodal scenarios. Third, (c3) many methods suffer from performance degradation when client datasets small. Finally, (c4) few approaches account for the progressive collection of new data over time. In this paper, we propose the PAC-PFL, framework to tackle these challenges (cl-c4).
Our proposed framework leverages probabilistic models to address (cl) by accounting for uncertainly. In the considered setup, each client places a prior distribution over models and updates it based on its local data to derive a posterior distribution. A naive PFL approach is to collaboratively learn a shared prior distribution and treat posterior inference the personalization step. However, this method conflicts with the Bayesian framework since the learned prior depends on each client's data (Box & Tiao 1992), as demonstrated in Figure I We overcome this by leveraging PAC-Bayesian inference, which accommodates data-dependent priors Rivasplata et al. 2020). Additionally, to address heterogencity (c2), we introduce a novel inference approach: instead of relying on a single shared prior, we learn a hypeer-posterior distribution over prior distributions from which clients sample their priors. By decoupling clients' posterior distributions from a single shared prior, our framework enhances adaptability to diverse data distributions.
Low-data scenarios (c3) are prone to overfitting, where models exhibit strong performance on training data but generalize poorly to unseen samples. PAC-PFL addresscs this challenge by selecting hyper-posterior that minimizes bound the generalization error. As shown in Section this approach introduces a principled regularization mechanism for the poseerposterrio, enabling the framework to learn complex models while mitigating the risk of overfitting. Regarding (c4), PAC-PFL accommodates collecting new data over time. In practice, only a subset of clients may communicate with the server during any given iteration, and these clients may gather new data locally before communicating again with the server. Such data can be utilized for personalization but not for updating the shared hyper-posterior. Furthermore, PAC-PFL supports new litents-devicee joining the system later that have never communicated with the server. A distinct generalization bound applies to this group; underscoring the importance of differentiat bet summarizes the introduced challenges (cl-c4) and how they are addressed both in theory and through experimental validation.
existing and new clients. Table [
We evaluate PAC-PFL on Gaussian Process (GP) regression and Bayesian Neural Network (BNN) classi- fication as representative examples of probabilistic models. Our experiments demonstrate that PAC-PFI. yields accurate and well-calibrated predictions (cl), even in highly heterogeneous (c2) and data-poor (c3) scenarios. Furthermore, we illustrate that PAC-PFL facilitates positive transfer learning from existing to new clients (c4). We empirically showcase the effectiveness of learning hyper-posterior, instead of single prior, in highly heterogencous cases. Lastly, we provide an interpretation of our method through Jaynes' principle of maximum entropy (Jaynes 1957 in Appendix 8.1
2 Related work and novel contributions
Meta-PFL. PAC-PFL belongs to the meta-PFL category (Kulkanni et al. 2020) Meta-learning, developed independently of FL. involves training a global model on various related learning problems (tasks), which can be efficiently fine-tuned for a new task. The link between meta-learning and PFL was first explored in Jiang et al. (2019), where they proposed simultaneously learning the global model and its personalization. Other approaches have investigated learning global model initialization that performs well after being personalized by individual clients KKodaak et al.) 2019 Fallah et al. 2020 Chen et al. 2018). In Fallah et al. (2020), a scalable algorithm, called MAML, is proposed that limits the personalization step to one or a few gradient descent steps. However, when clients have small datasets, the personalized models tend to remain close to the shared initialization, resulting in a strong resemblance between them. Consequently, these methods may lack the necessary personalization capability in highly heterogeneous cases with small client datasets. Additionally, all these methods are frequentist, specific to parametric models, and lack generalization bounds.
2

---

Published in Transactions on Machine Learning Research (02/202))
federated learning
personalizing
Sn
SJ
optimal hyper-posterion
optimal posterior model
hyper-prior
Corollary
semple
prior
Corollary 3.4
4.4
P
Qi
Figure 1: Illustration of the proposed PAC-PFL framework. For given hyper-prior distribution P, the server computes the optimal hyper-posterior Q' as per Corollary 4.4 through communication with clients owning datasets Si,..., S,. During personalization, each client i draws a prior distribution P from 2 combining it with its local dataset S. and potentially new data S, to derive the optimal posterior distribution Q according to Corollary 3.4 The client then samples a model from Q; for making predictions. Note that the prior P depends on each client's data, S,, conflicting with the Bayesian framework and necessitating proper consideration, as discussed in Section 3
Table 1: Challenges, solutions proposed by PAC-PFL, and datasets representing them. The number after each dataset name indicates the number of data samples per client. Detailed descriptions of these datasets can be found in Appendix 8.5
Challenge
Solution
Dataset
quantify uncertainty
probabilistic modeling
high heterogeneily
formulate a hyper-posterior minimize generalization bound
PV-EW (150), PV-EW (610)
overfitting
PV-EW (150), PV-S (150), FEMNIST (20)
new data
incorporate personalization
PV-EW (150), PV-EW (610), PV-S (150), PV-S (610)
FL for probabilistic models. Several FL methods aim to learn a global postcrior without personaliza- tion (Bui et al. 2018 Al-Shedival el al.] 2020 Kassab & Simeone 2022). Learning personalized posteriors is studied in Corinzia & Buhmann (2019), where the posterior for each client is decomposed into global and per-client variational factors. However, this method encounters scalability challenges in systems with nunerous clients, personalized models are refined sequentially. As mentioned in Section [ an alternative approach is to learn shared prior and consider posterior inference the personalizen lersonalization step. This idea is adopted in methods like pFedBayes (Zhang et al. 2022) and pFedGP (Achituve et al.) 2021). In paed clients compute personalized posteriors by optimizing their data likelihood while adding regularization to- wards the global prior. The server calculates the prior by minimizing the average client loss. Due to relying on bi-level optimization, payd imposes a significant computational burden.
The closest work to our approach is pFedGP which learns global GP prior by maximizing the average Log Marginal Likelihood (LML) across all clients. Consequently, clients perform posterior inference for personalization. However, pFedGP only focuses on learning the covariance of the prior and sets the prior mean to zero. The reason for ignoring the mean may be that the LML implicitly regularizes the covariance, while the mean can quickly overfit (Fortuin et al.) 2020). Both PFedBayes and pFedGP may exhibit limited personalization flexibility, as they utilize the same prior distribution for all clients. Additionally, their inference procedures are heuristic due to deviating from the Bayesian setup by using data-dependent priors.
PAC-Baycsian meta-learning. The setup described in Section 1 is rolated to meta-lcarning within the PAC-Bayesian framework (Amit & Meir 2018 Pentina & Lampert, 2014 Rothfuss el al.] (021). In this scheme, a meta-learner is presented with a sequence of heterogeneous learning tasks, corresponding to existing clients in PFL, and aims facilitate posterior inference for an unscen task, corresponding a new client. The meta-learner employs the previous tasks' data to learn a hyper-posterior distribution that leads to good generalization to the new task, establishing data flow from previous tasks to the task.
3

---

Published in Transactions on Machine Learning Research (02/2025)
PAC-Bayesian federated-learning A PAC-Bayesian PFL algorithm introduced by Jobic et al. (2023) optimizes both the parameters of a shared prior distribution and personalized posterior distributions by minimizing non-vacuous generalization bounds from McAllester (2003). While PAC-PEL accommodates arbitrary distributions, this method assumes Gaussian prior and posterior distributions. To avoid data- dependent priors, splits each client's dataset: one half to learn the prior and the other to learn the posterior, sacrificing data efficiency. This approach evaluates the prior based on the loss incurred when sampling models directly from it. However, an alternative strategy would be to evaluate the loss of models drawn from the posterior distribution corresponding to this prior, potentially enhancing performance by directly optimizing the effectiveness of personalized posteriors.
Novelties We propose PAC-PFI., a probabilistic PFL framework that addresses challenges (c1)-(c4) while introducing two significant advancements over existing methods. First, PAC-PFL improves upon Bayesian and PAC-Bayesian PFL approaches by offering a theoretically rigorous and data-efficient inference pipeline that eliminates the need for data splitting. This is achieved through PAC-Bayesian methods allowing for data-dependent priors that are differentially private (Dziugaite & Roy 2018). along with an analysis of the proposed method's privacy properties. Second, unlike PAC-Bayesian mela-learning approaches that primarily benefit new clients (tasks), PAC-PEL incentivizes existing clients to collaborate in the framework by learning a hyper-posterior distribution tailored for their data. To achieve this, we define a novel loss function that evaluates the Figure D for existing clients. We then derive a generalization bound for this loss function, focusing on unseen data from existing clients rather than new clients, as in meta-learning. With these advancements, PAC-PEL provides a principled solution for addressing existing challenges in PFL.
based on the accuracy of models derived via the pipeline in
3 Preliminaries and notation
We start by presenting the notation and key concepts. A summary of the introduced notation is provided in Table 4 Consider set of n E N clients, referred to as eristing clients, each observing samples of feature- target pairs (x, v), where x E Rx C Rdd represents the features and y E Ry denotes the target values. The feature space Rx and the target space Ry are identical across clients, and for simplicity, we assume the target y is scalar, i.c., R, C R. Each client has access to two distinct i..dd datasets: S, ~ D.T containing m, e N samples, and S; Di containing m, E N U {0} samples. The first dataset, S,, is non-empty and used by client during both FL and personalization phases. The second datasct, S,, consists of samples acquired later, which are used exclusively for the personalization phase and may be empty. We assume that m; S m, for all clients i, and we denote the mumber of clients for which m. > 0 as n2 E {O......
The data held by each client is sampled from an unknown data distribution D; over the support R₂ = Rx x Rg. To account for client heterogeneity, we allow both the distributions and the number of samples to vary across clients, i.e., D; * DD and m; # my- We capture the relatedness among clients through a distribution T over the data distributions and sample sizes, such that (D., m4) ~ T. For convenience, we introduce the following notations: S == { D [Dr)
[m,,".. mnn) and m- = [m .m,]-
Each client i aims to learn hypothesis function h4 Rx + R2 for predicting the label y. of an unseen input x.. The error incurred by h; at (X,,y)) is measured by a loss function which we assume to be bounded, L: H Rg + [a, bj. Since the data distribution, D, is unknown, standard methods select the best hypothesis ac- cording to the set of observed samples, S,US; for instance, by minimizing the empirical risk associated with f.
Using a single hypothesis based on limited observations leads to epistemic uncertainty (Draper 1995 and overconfident target estimation (Kass& & Raftery 1995). To address this issue, the PAC-Bayesian framework [McAllester] 1999) places probability distributions over the hypothesis space, H. and combines all possible hypotheses sampled from these distributions for making inferences. Two such distributions are enlisted: a prior distribution P, and a posterior distribution Qi which depends on the prior and the observations through a mapping, Q, = Q(P,S, U S.). While the prior and posterior terms resemble the Bayesian terminology, the posterior mapping, Q, is not (necessarily) obtained through Bayes' theorem. Additionally, recent advances in PAC-Bayesian analysis enable employing priors that slightly depend on the data to make the learning pipeline more data-driven, which is contrary to the Bayesian formalism (Rvaaslaaaa et al. 2020).
4

---

Published in Transactions on Machine Learning Research (02/2025)
The flexibility in choosing the mapping Q can be exploited to achieve desired characteristios. Ideally, clients aim to choose Q that minimizes the true risk,
Client-level PAC-Bayesian bounds.
(Q(P))) - [e(h, z)],
(1)
where superscript C' indicates that L is calculated by client. In most cases, Dj is unknown and the true loss is approximated by its empirical counterpart,
( = Bn-QQ(P.(U5.) my + mh ZES,US,
Σ z).
(2)
PAC-Bayes theory (McAllestor 1999) provides a guarantec on the worst loss clients may suffer by upper bounding the unknown true risk based on the empirical risk in (2). The original PAC bound by McAllester (1999) assumes that the prior, P, is independent of the data. However, our setup, the prior is obtained from the FL algorithm acting on the data subset S,, hence, is data ditaadeeenddent. Dziugaite & Roy (2018) propose recipe for adapting any PAC bound with data-free prior to data-dependent prior that is stable to slight changes in the data, formalized by the notion of Differential Privacy (DP) defined below.
Definition 3.1 (dDoork& Roth 2014)). Let e E R4 and be a rondomized algorithm that generates a stochastic output given an input datasct. The algorithm A preserves G-DP if for all datascts S, and S; that differ in single data sample and all subsets O of possible outcomes of A,
- S Pr[A(S.) E EO] Pr(A(S.) 0j Se9
(3)
where the probability is w.r.t the algorithm's randomness.
Intuitively,
3 bounds the stability A to changing a single sample of S @work et al. 2015 The privacy/stability level controlled by e: for smaller .A more private/stable.
To state PAC bound for client i, regard the federated pipeline as randomized algorithm that takes S; as input and outputs P Asiss (S.). The subscript S) contains samples from all clients except and emphasizes that P depends on data from other clients likewise. However, DP is only studied for the argument, Sy. Randomization in Asis, arises from sampling the prior from the hyper-posterior.
Assune the prior obtained by A preserving e-DP. We apply the method Dziugaite Roy 2018 on a bound due to Alquier al. (2016) to derive the following theorem.
Theorem 3.2. Fir a data- dependenn prior P obtained by an 4-DP algorithm, a data distribution D, and a bounnedd loss function E(.,-) E [a, b]. For every B > 0, confidence level 8 E (0, 1], and posterior Q. = U(P)), the inequality
+ (KLQ.IP)) + B²(b - a)² + ((E,, m., 8) In ( 8(m; + m.)
(.). < ( +
(4)
holds with probability at least - over S, ~ D. and 8, ~ D. In the above, [(,, m, 6) = 0.5m, + 6, 0.5mm, Inn4),)+ and does not depend on the posterior. is assumed that the KL divergence between Q; and P erists and is denoted by (L().))))
Remark 3.3. The term I(G, mi, is the only difference with the bound of Alquier et al. 2016) with data-free prior.
We refer to Theorem 3.2 as the client-level bound.
Optimal posterior. The bound in 4 holds for all Q and thus can be minimized w.r.t Q, to obtain the tightest upper bound. Since & only reflects through the term in (4, the optimal posterior is the same the minimizer of the bound with data-free prior derived in Catoni 2007
5

---

Published in Transactions on Machine Learning Research (02/2025)
Corollary 3.4 (Catoni 2007 ). Given prior, P, obtained through an cpDDP algorithm and observations, S; U.S.S. USS the oplimal posterior minimizing the right-hand side of (4) is the Gibbs distribution:
-8 Q7(h) P(h) mI Fii ExEsJus, ((h,2))
(5)
(P,S
(P,S. RRape Dresjue, ((h,2))
where 23 (P is normalization constant. We denote the dependence of the oplimal posterior on the prior and data using the operator Q": Q = (P,
With m, + m, and the negative log-likelihood loss, (hh, = - Pr/s//,,, zÿ and Q; simplify to the LML and the Bayes posterior respectively (Guedj 2019).
Plugging in the closed-fore formula of the optimal posterior into the client-level bound obtains:
((((()..U.))) 1 (
BB a)² 8(m; m.) + I(e,, m,, 6) In (
(6)
holding with probability at least 1 over S; D. ~ D. The simplified bound 6
removes the
explicit dependence on Qi and is tighter than the generic bound per
In the rest of this paper, assume that clients utilize O whenever the privacy requirement of Theorem 3.2 is satisfied. To highlight the generality of our approach, we note that Bayesian inference is subcase of the assumed setup.
4 Theoretical framework for PAC-Bayesian federated learning
In this section, we present data-driven approach for obtaining prior distribution. Rather than focusing on a single prior, we propose learning distributions over priors, enabling a more flexible and robust framework. To formalize this, we introduce the following key concepts:
Definition 4.1 distir-distiibutions). A hyper-prior, P, is a distribution over prior distributions that is independent of clients' datascts. Converscly, a hyper-posterior, Q, distribution over priors that can rely the data of existing clients, Si,..., ,...
A trusted server communicatos with the cxisting clients to extract common knowledge in the form of hyper- posterior distribution without directly accessing their datasets (as enforced by FL). At each communication round, the server sends the hyper-posterior to the clients. The clients proceed by repeatedly drawing a prior from the received hyper-posterior and using it to calculate the optimal posterior per (5), involving any potential additional samples, S, in the inference procedure. The goal is to find the optimal hyper-posterior, O', such that. the posterior obtained through the described pipeline has an average low true risk (1) for all clients. An overview of the setup depicted in Figure
In the sequel, consider hyper- posterioss Q, that satisfy the condition that for finite E R,, sampling P from Q preserves e-DP for all clients. This assumption enables us to use Corollary 3.4 and is rather weak
as can be arbitrarily, alboit not infinitely, large. Analogous to the procedure in Section 3 we establish PAC bounds for the pair P and @. Next, we derive the closed-form formula for the optimal hyper-posterior, Q' which minimizes the PAC bound, and verify that Q" satisfies the privacy assumption outlined above. Finally, we establish a PAC bound for new clients who sample their priors from Q" without participating the federated learning process. An interpretation through the principle of maximum entropy (Jaynes 1957) and all proofs provided in Appendices 8.1 and 8.2 respectively.

---

Published in Transactions on Machine Learning Research (02/2025)
Server-level PAC-Bayesian bound. Following the introduced inference setup, we evaluate the quality of hyper-posterior distribution
Q using the server-leuel true risk, defined as:
2 (Q,D,S, = Σ Ep Ep-eB B... 2 (Q),
(7)
This metric computes the average true loss across all clients, considering all possible sets of additional samples, S, of size m. However, the server-level true risk is intractable due to its reliance expectations over the underlying data distributions D. To address this, we approximate it with an empirical estimate:
n (Q, = Σ ( ((P..)(())P.S))
where observed samples replace the unobserved future data. We refer to
S as the server-level empirical loss.
Below, we present our first main contribution, which is a PAC bound on server-level loss. Theorem 4.2. Let e(:--) ) [a, b] be a bounded loss. Define:
A
2Am; 1 min {b- b(e mi thg n
(b-a)
-2800 (b- m 1in
(b-a)))
for all i E n.. Assume clients employ the optimal posterior with parameter B 2 1/n. Lel Q be a super-posterior such that sampling P from Q preserves e-DP for all clients. For every hyper-price P independent from S, 0, >> n2 + v, and confidence level 8 (0,1),
(Q, m)
Ep (P, +
1
+
122 +
KL(2)P)
nB
nB
X
B(b
n a)²
Σ 1=1
3Σ A
1
In )
(9)
m
8(nz + +2)
holds with probability at least 1 8 over S D and S; ~ D for 1,... n. The constant is chosen to be very small and avoids namerical issues when 0.
The first term the right-hand side of (9) falls within [a, b] due to our bounded loss assumption. The summand 1n z (P, in this expression decreases when the prior P "aligns better" with the dataset S₁, favoring hypotheses with lower empirical costs over S, Consequently the first term the right-hand side decreases if Q assigns higher probability to priors that are, on average, "better aligned" with all clients' data. The KL-divergence term in (9) regularizes the hyper-postcrior towards the hyper-prior and avoids overfitting to the existing clients when n is small. The bound depends on the number of clients and the available dataset sizes, n and m. The bound (9) tightens with increasing the number of samples involved in calculating the empirical loss, mi demonstrating consistency with the PAC-Baaes framework. Finally, the bound relics predictions for the number of clients with new samples and the corresponding new sample sizes, n2 and m. While the server knows and m, the estimates for n2 and m might be coarse. The next lemma states that a pessimistic forecast leads to a looser upper bound.
Lemma 4.3. If the number of samples of client i, m; 2 0, is wnknoum, Theorem 4.2 holds when replacing A with (b-a)/n and counting client i in n2, i.c., if m 0.
The asymptotic behavior and von-vacuousness of the client and server-level bounds are addressed in Ap- pendix 8.3.1
Optimal hyper-posterior. Our algorithm picks the optimal hyper-posterior, º* leading to the lowest upper bound on the server-level true risk per 4 Inspecting the structural similarity between the server and client-level bounds in and (9), arrive at closed-form formula. for Q*.
7

---

Published in Transactions on Machine Learning Research (02/2025)
Corollary 4.4. When clients use the optimal posterior, Qi the optimal hyper-posterior is Gibbs distribu- lion with parameter X/ Bm(n2 + v)):
exp T n
In (28(P.5.))
where (P) = Eppp exp Ei= In (P) (n is a normalization constant.
The parameter depends on the number of clients, n and u2 but not on the number of samples, m and m. If ng unknown, it can be replaced consistently with Lemma In this case, looser upper bound would be minimized.
The privacy of sampling prior from Q" is crucial to obtain & for plugging it into the client-level bound (6) We rely on a result by Mir 2012) that proves the DP of sampling from the
and for employing Theorem 4.2
Gibbs distribution.
Lemma 4.5. A prior sampled from Q* preserves DDDP for client i, where 5, = 23r a)/ms.
As a result, 2 satisfies the privacy assumption of Theorem 4.2 with 6 maxie(1.... in Additional insights into the role of DP in our framework are provided in Appendix 8.4
PAC-Bayesian bound for new clients. So far, we considered a fixed set of cristing clients who partic- ipate in training the optimal hyper-posterior. In realistic FL setup, there might. be new clients (see e4) who join the system later and hence, do not engage in federated training. A new client holds a presumably small set of samples which leads to overfitting. Assuming the existing and new clients are similar, it is constructive for the new clients to readily employ Q* without having contributed to training it. In this section, we establish a PAC bound for such new clients.
Section 3 introduced the distribution T to capture the similarity among existing clients. Consistently, we expect that new client sampled from the same distribution, (D., m.) T. In line with previous notation, S, Dim, is dataset of size m, employed by client for personalization but excluded from FL. Our second PAC-Bayesian bound applies to new clients and is presented below.
Lemma 4.6. For new client sampled from T adopting Q" and Q" as per Corollaries 3.4 with probability at least 1 - 6 over (Di, m.,) TT and S. ~ D that:
it holds
E.D. C
<
+
+
In (P,S)
n3
X
a)²
11
1
In 1 1n(3).
+
8n
m
+
Vn
Since Q* is tailored to minimize the server-level bound for existing clients, the bound in Lemma 4.6 is looser than that of Theorem 4.2 with Q = Q* (proof in Appendix 8.2.5). This motivates the clients to actively engage in training º" rather than readily employing the learned hyper-posterior.
5 Practical federated implementation
Section 3 justified learning a distribution over priors for heterogeneous clients, recognizing that selecting a single best prior may not accurate or feasible. Consequently, Corollary [.4] provided the optimal hyper- posterior. This section tackles computational constraints at both the client and server levels and introduces practical PFL algorithm.
Calculating the LML at the client level. The formula for Q* in Corollary 4.4 relies on (P, for (0....,))) As per Corollary 3.4 calculating (P, entails computing the expectation over all hypotheses sampled from the prior, which is generally intractable. For the negative logllikelihood loss, we set B m. + m making Zg align with the LMI., as discussed in Section 3 We calculate the LMT. for two
8

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm PAC-PFL executed by the server
Input: number of SVGD priors k, hyper-prior P, parameter T, number of iterations T, number of clients per iteration e, mini-batch size b, learning rate 7
2: Initialize priors P......, .................... iid pk
Iniliolize
for t to T do
Select random subset C. of с clients
5: for cach selected client i in C. in parallel do
6:
G; + Client Updale(b, $1,...,
b Collect client updates
7:
G 1/2 Eiec G.
Aggregate client updates
for to do
VΦ. In 2¹ (*x + Va. In P()x)))) G
Gxa is the k-th yow of G
10:
for R = to do
ΦR t Φx + + (ksvGD(0I,0)) Vn In (+b) + V ksvGD(dr, Φ.))
11:
> SVGD update
12: return Po...... Poc
SVGD approrimation of Q"
Algorithm Client Updote for client with dataset S,
1: Input: mini-batch size b, current particles $1,... Φx
Sample mini-batch s of size b from S
Data subsampling
for == to do Compute V In z (Po.. s(t)) through automatic diffcrentiation the LML, given by 13 and 38
G a In n S S(1)), V In De . )!
6: return G;
Update from client
scenarios: clients using GPs or BNNs. The LML is available in closed form for GPs but intractable for BNNs. To address this, we use the approximation method described by Rothfuss al. (2021). The formulas for calculating the LML are provided in Appendices 8.3.2 and 8.3.3
SVGD the server level. Given (P, Q* is computable up the constant (P,P,S) which Stein Variational Gradient
leaves sampling from Q" intractable. Following Rothfuss et al. (2021), Descent (SVGD) [Liu & Wang) 2016) that approximates Q" set of particles, P Po.. Each particle Po. is prior parameterized by Φr. SVGD is initialized with a set of priors and then iteratively transports them to match Q*. This is achieved through a form of functional gradient descent on the SVGD loss (see Appendix 8.3.4. making it suitable for being integrated into an FL scheme.
As SVGD deterministic [Liu 2017 the inherent privacy of Q" established Lemma 4.5 compromised. To reintroduce privacy, a conventional approach involves injecting noise into the SVGD gradients (Geyer et al. 2017 We propose privacy-preserving variant of PAC-PFL in Appendix 8.4
Federated algorithm. The pseudocode of our algorithm is presented in Algorithm [ Initially, the server samples ....... Pos, from P, defined zero-mean multivariate Gaussian distribution with diagonal covariance matrix. At each iteration, the server randomly selects subset of existing clients and sends them $1...., Φz. The selected clicnts perform the Client_ Update sub-routine to compute the gradient of the LML with respect to the particles on mini-batch of size of their data and send it back to the server. The server updates the particles based on the aggregated gradients, G, and the learning rate, The particle update formula also depends on the SVGD kernel, ksvGD, as discussed in Appendix 833.4 For comprehensive list of parameters used in our theoretical results and algorithm, along with selection guidelines, please refer to Appendix 8.3.5

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Summary of the employed datasets. The number of samples is the dataset size per client.
Dataset
Num samples (m)
Num clients (n)
Task
PV-EW (150 610)
150 610
24
regression
PV-S (150 / 610)
150 610
24
regression
FEMNIST (20 /500)
20 ~ 500
40
10-way classification
EMNIST
E [516, 1954]
80
62-way classification
Polynomial
10
24
regression
6 Experiments
We evaluate PAC-PEL on four datasets: photovoltaic (PV) panels and Polynomial datasets for regression, alongside FEMNIST (Caldas et al. 2019) and EMNIST (Cohen et al. 2017) datasets for classification. Our algorithm consistently outperforms federated and data-centric basclines, improving the prediction accuracy and the calibration of uncertainty estimates simultaneously. These enhancements are evident in reducing the variance and mean of these metrics across existing and new clients. Our experiments demonstrate the effectiveness of the solutions proposed in Table for mitigating the identified challenges.
Datasets. The PV dataset comprises PV generation time-series data from multiple houses within a city, with each house treated client.. The clients exhibit heterogeneity due to variations in location, shadows, and orientation relative to the sum (c2). We explore two scenarios: PV-EW, featuring bimodal distribution over clients, where half are oriented almost eastward and half almost westward, and PV-S, with all clients oriented almost southward. In both scenarios, we consider 24 existing clients and 24 new clients (c4) and examine m, = 150 or m, = 610 training samples per client. We specify m, in front of the dataset name, such as PV-EW (150). The case with m, = 150 imposes (c3). More information about the PV dataset and the description of the Polynomial dataset are available in Appendix 8.5
The FEMNIST dataset consists of handwritten characters from various writers, treated as clients, and we employ it for 10-way digit classification. Heterogencity arises from distinct handwriting (c2). We select 40 clients and examine two scenarios: FFMNIST (20) with 20 and FEMNIST (500) with an average of 500 samples per client. In the low-data case (c3), we demonstrate PAC-PFL's superior performance over all basclines. In the full-data case, we highlight the scalability of our algorithm with large datasets. The EMNIST dataset is detailed Appendix 8.5
Table 2 provides summary of the employed dalasets.
Baselines. We examine two probabilistic PFL methods, pFedGP (Achituve et al, 2021) and payd Zhang et al. 2022), and two frequentist PFL methods, MAML (Fallah al. 2020 and MTL (Evgeniou & Pontil 2004). Additionally, we consider two non-fedeed nen-eedeeteed approaches: Vanilla, where each client trains model individually, and Pooled, where single model is trained in a data-centric manner a pooled dataset comprising all clients' data. The Pooled approach is expected to perform poorly for heterogeneous clients due to the lack of personalization.
Syper-parameters for each method are tuned using vrldsvvali Further baseline details can be found in Appendix 8.6
Model configuration For the PV experiment, we train Neural Network (NN) using the MAML and MTL methods, and GP using the PAC-PFL, pFedGP, Vanilla, and Pooled methods. Inspired by Fortuin et al. (2020): Rothfuss al. (2021) we parameterize the GP mean and kernel with two deep NNs and consider Gaussian likelihood. This model enhances the expressive power and scalability of GPs to high-dimensional data Wiilson et al. 2016). Further details are provided in Appendix 8.3.2
For all classification experiments, we utilize a Bayesian Convolutional Neural Network (BCNN) (Gal & Ghahramani] 2016) for the Bayesian approaches and Convolutional Neural Network (CNN) for the fire- quentist methods. All BCNNs and CNNs share the same architecture proposed by Zhang al. (2023) Specifically, the network consists of two convolutional layers with 5 x kernels, ReLU activation functions.
10

---

Published in Transactions on Machine Learning Research (02/2023)
Client type:
existing
1.0
0.15
0.8
RSMSE
0.10
0.6
0.05
0.4
Vauilla
MAML
MTL
pFedGP
PAC-PPL
Vanilla
pFodGP
PAC-PFL
Figure 2: Box plots of test RSMSE and CE for existing and new clients in the PV-EW (150) dataset. The line within cach box is the median. PAC-PFL excels in CE median, CE spread, and RSMSE median over baselines. RSMSE spread is comparable to MTL and pPedGP. Pooled GP results are not plotted due to poor performance but are reported in Appendix 8.6
max pooling, and 10 and 62 output channels, respectively. These convolutional layers are followed by two dense layers with ReLU and SoftMax activation functions.
Metrics. We assess prediction accuracy and calibration for each client. For regression, we root standardized meon squared error (RSMSF) that normalizes RMSE by the standard deviation of targets. For classification, we measure the percentage of correctly classified samples. Additionally, we compute the calibration error (CE), which quantifics the deviation of predicted confidence intervals from actual proportions of Lest. data within those intervals (Kuleshov el. al. 2018), using the formula in Rothfuss et al.) (2021) (sce 8.6.1. CE applics exclusively probabilistic models and is irrelevant to frequentist basclines, such MAML and MTL. In evaluating FL methods, the sample mean of metric is often biased estimate due to correlations across clients. We employ box plots to analyze the distribution of metric across clients, offering a more comprehensive understanding.
Results. Figure 2 presents the results for PV-EW (150), highlighting several observations. The limited sample size adversely impacts the local Vanilla GP (c3). MAML does not perform well, likely because single gradient descent step lacks the necessary personalization (c2). Notably, PAC-PFL, outperforms all baselines in terms of prediction accuracy and calibration (see cl) due to its strong personalization capability, addressing (c2), and inherent regularization, resolving (c3). Moreover, it exhibits the best generalization to new clients, overcoming (c4). The results for other regression datasets are available in Appendix 8.6
Test accuracy and CE for existing clients in the classification datasets are reported in Table 3 Remarkably, PAC-PFT. outperforms other baselines the FEMNIST (20) and EMNIST datasets. On FEMNIST (500). PFedGP demonstrates a slight advantage in terms of the mean, albeit with considerably higher standard deviation, indicating sensitivity to initialization. Given the closely aligned means for PFedGP and PAC-PFL and the fact that the confidence interval of PAC-PFI. is encompassed within that of pFedGP, PAC-PFL, is a more reliable method. Additionally, pFedGP has substantially higher computational cost than PAC-PFI Therefore, we conclude that PAC-PFL the superior choice for all datasels.
Achituve al. 2202 propose alternative variants of pFedGP that trade off accuracy for reduced computational cost. However, employ the original algorithm. *CE calculated for pFedBayes the available software only provides prediction and lacks variances required CE computation.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Table Comparison of probabilistic ( and non-probabílistic FL, approaches along with probabilistic non-foderated basclines on classification tasks. Average test accuracy (%) for all basclines over trials reported, where + captures
and calibration error (CE) for probabilistic approaches
95% confidence interval. Best results in each column are marked.
Dataset
FEMNIST (20)
FEMNIST (500)
EMNIST
Metric
Accuracy
CE
Accuracy
CE
Accuracy
CE
PAC-PFL
94.2 + 2.6
0.08 = 0.01
97.1 + 1.6
0.05 + 0.01
87.1
0.04 0.01
pFedGP
83.6 + 2.0
0.10 = 0.04
97.7 = 6.6
0.02 0.01
82,4 1.0
0.06 0.02
paedBayes
87.0 + 2.0
88.3 + 2.4
79.0 1.3
FedAvg
88.1 + 1.7
96.7 + 0.5
79.9 + 1.0
MTL
75.8 + 3.7
87.5 = 0.1
78.2 1.0
MAML.
82.0 + 6.7
88.1 + 2.6
$1.5
Vanilla
81.9 + 1.3
0.12 0.01
92.5 + 1.8
0.33 + 0.06
71.0 1.2
0.09 0.01
Pooled
89.4 + 4.3
0.11 0.05
94.3 = 2.1
0.07 + 0.03
63.8 = 2.9
0.04 0.02
PV-EW (150)
PV-S (150)
0.43
RSMSE
0.42
0.41
2
16
1
2
4 8
16
number of particles
number of particles
Figure 3: Ablation study on the impact of the number of SVGD particles, k, on RSMSE of the existing clients in PV-EW (150) and PV-S (150) datascts. Each experiment is repeated over 5 random seeds. The error bars correspond to the mean + standard deviation. Computational cost scales linearly with k (see Appendix 8.3.6
Ablation study on k. Figure illustrates the impact of the number of SVGD particles, h, on RSMSE for existing clients in the PV-EW (150) and PV-S (150) datasets. In both datasets, increasing & enhances the SVGD approximation and the overall performance. The improvement particularly pronounced PV-EW due to its higher heterogeneity level and bimodal nature. Notably, transitioning from one to two particles leads significant performance boost, as single particle fails capture patterns in both modes. This highlights the efficacy of learning hoper-posterior inslead of global prior, corresponding to k 1.
7 Conclusion
This paper presents PAC-PFL, novel PFL algorithm that enables the learning of probabilistic models. The proposed approach learns shared hyper-posterior in foderated manner, which clients use to sample their priors for personalized posterior inference. To prevent overfitting, PAC-PFL minimizes an upper bound on the true risk the clients participating in federated training. Moreover, the learned hyper-posterior can applied to new clients who did not participate the training, resulting in positive transfer. Conducting experiments on several heterogeneous datascts for regression and classification, we empirically demonstrate that PAC-PFT. produces accurate and well-calibrated predictions.
12

---

Published in Transactions on Machine Learning Research (02/2025)
There are two main directions for future research: improving client-level computational complexity (detailed in Appendix [83.6] and addressing the privacy-utility trade-off more effectively. Our framework leverages DP to derive valid generalization bounds despite having data-dependent priors and to avoid data leakage, as typical in FL. While our thcoretical results in Section f show that our ideal pipeline provides DP, we Torfeit this properly due to the SVGD approximation technique. DP can be reintroduced leakage using the common method of injecting noise during training (Geyer et al. 2017 as demonstrated in Appendix 8.4 but this may compromise accuracy (Bagdasaryan al. 2019 Exploring alternative privacy techniques is avenue for future research.
prevent data
Acknowledgments
This work was supported as a part of NCCR Automation, National Centre of Competence in Rescarch, funded by the Swiss National Science Foundation (grant number 51NF40 2_211____111 and the NECON project (grant number 200021219431).
13

---

Published in Transactions on Machine Learning Research (02/2025)
References
Photovoltaic geographical information system (PVGIS) of the joint research center of the European commis- sion. http::/:pt....aaaeeeee.eeeee.eep
[Accessed: 17 May 2023].
Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with Gaussian processes. In Advances in Newral Information Processing Systems, volume 34, pp. 8392 8406, Curran Associates, Inc., 2021. URL httpt: https:///roceeedgsss neurips cecpipp..cppapee cw/paper_aaaeeeeeeeeeeeee filaap/rffee////// files/ optp://00f//11kkkoooo//ttt116660
Maruan Al-Shedivat, Jemnifer Gillenwater, Eric P. Xing, and Afshin Rostamizadeh. Federated learning via posterior averaging: A new perspective and practical algorithms. CoRR, abs/2010.05273,, 2020. URL
https://doi.. org/ 20550//rrivv20100 201.xx....0000273 05273
Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of Gibbs posteriors. Journal of Machine Learning Research, 17(236):1 41, 2016. URL http://jmlr: org/
papers/v17/15-2999 29mmm
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended PAC-Bayes theory. In Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 203-214 PMLR, 10-15 Jul 2018. URL https://proceeddngss mlr wwwpsemwww......eeeeeeeeeeeeeeee press/v80/ amitiBa.htam
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate im- pact on model accuracy. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/ paf
wttpppp.ccpppppereee httpp/aaeeaeepppperrrrraae ftlles/pappr//211//iee
George E.P. Box and George C. Tiao. Nature of Boyesian Inference, chapter pp. 1-75. John Wiley and Sons, Ltd, 1992. URL https:: /ttpps://onlinellrrrrr wwl.lee..cma/ao///////000000///0
Thang D Bui, Cuong V Nguyen, Siddharth Swaroop, and Richard E Turner. Partitioned variational infooomaniiffeeeeeo inference: A unified framework encompassing federated and continual learning. arXiv preprint arXiu:111 11206, 2018. URL httthtttt://oiii htttp:://aa..orgg/0.446665 httpg//.1..45//2/1i1... 1811.11206
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Koncčný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. LEAF: benchmark for federated settings, 2019.
Olivier Catoni. PAC-Bayesian supervised classification: The thermodynamics of statistical learning. IMS Lecture Notes Monograph Series, 56:1-163, 2007. URL httpp:://dt/ wtt.://///..111/////770000000000
Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding. pFL-bench: a comprehensive benchmark for personalized federated learning. In Proceedings of the 36th International Conference on Newral Information Processing Systems, 2024. ISBN 9781713871088.
Fei Chen, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning for recommendation. CoRR, abs/ Abs/1802.00886, 2018. URL https:://doi.rr/// 48550/arXiv 1802.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint. Conference on Neural Networks (IJCNN), pp. 2921- 2926, 2017. doi: 10.1109/ MAL109/1JCCNS LJCCN.2011.77966..
Luca Corinzia and Joachim M. Buhmann. Variational federated multi-task learning. CoRR, abs/1906.06268. 2019. URL https::///oo/ /ttp:://///oo.o/1////0 40w11////.550............ 1906. 06268
Thomas Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.
14

---

Published in Transactions on Machine Learning Research (02/2025)
David Draper. Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society. Series B (Methodologicnl), 57(1):45-99) 97, 1995. URL
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy, Found. Trends Theor. Comput. Sei., 9(3 4):211 407, aug 2014. URL http::///0ol.org//011156///////0
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Gen- eralization in adaptive data analysis and holdout reuse. In Proceedings of the 28th International Con- ference on Newral Information Processing Systems Volume 2, NIPS'15, pp. 2350-2358, Cambridge, MA, USA, 2015. MIT Press. URL htpps///orooosssennggg heweseeeurrppp. wwwrrps..c/papee.eeeeeeeeeeeeeee httppp/cccpppppfffeeeeeeeeeeeeee www.fflllppppeeeee pttss/pppr//2115.. //ahh// bad5737000412558884077707033333 Abstractttttttttttt..
Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent PAC-Bayes priors via differential privacy,
In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
pdf
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '04, pp. 109-117, New York, NY, USA, 2004. Association for Computing Machinery. URL https:///oi.onn org/ 10. 1044/10114455. 1014067
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Advances in Newral Information Processing Systems, volume 33, pp. 3557-3568, Curran Associates, Inc., 2020. URLatppss/proccddgg. neurips.
pdf
Vincent Fortuin, Heiko Strathmann, and Gunnar Rätsch. Meta-learning mean functions for Gaussian pro- cesses, 2020. URL htttps://doi.org//h 40t//000..555//rxxiv 1901. 1901.
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with Bernoulli approximate variational inference, 2016.
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXio:1712.075:: 2017. URL https://doi. org.0/0...r/..... 4055//0000000///// 448500/aXi11 1712. 07557
Benjamin Guedj. A primer PAC-Bayesian learning. 2nd Congress of the French Mathemaliiaa Sociely, 2019. URL
/tt////...rrr/100..8866////1 05353
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1336 PMLR, 06-11 Aug 2017. URL htttps://proceedigngg mlr. httes//7ogguo7a..tt
Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, (8(31)):1333) 30, 1963. URL http://www. jstor org/stablle/288252
Will Holmgren, Calama- Consulting, Cliff Hansen, Kevin Anderson, Mark Mikofski, Antonio Lorenzo, Uwe Krien, bmu, Adam R. Jensen, Cameron Stark, Anton Driesse, DaCoEx, Miguel Sánchez de León Peque, kt. mayudong, Naman Priyadarshi, Heliolytics, Ed Miller, Mare A. Anoma, Veronica Guo, Leland Boeman, Joshua Stein, Will Vining, jforbess, Tanguy Lunel, Alexander Morgan, Joe Ranalli, Carlosbogo, Cedric Leroy, and Ahan M R. pylib//vliibpyyhhee v0.9.3, September 2022. URL hhttps://dot http:///ooooo//../////// Renodo.708333
E. T. Jaynes. Information theory and statistical mechanics. Phys. Rev., 106:620 630, May 1957. URL
https://link. aps. .pt. wwtp/////////oooooo httg//bbi/10.103//hhyeerr 106. 620
15

---

Published in Transactions on Machine Learning Research (02/2025)
J. L. W. Jensen. Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acto Mathematica, 30(none):178 193, 1906. URL. htttps:///bi.org/// 1tt.//0000100///24418850
Yihan Jiang, Jakub Koncčný, Keith Rush, and Sreeram Kannan. Improving federated learning personaliza- tion via model agnostic meta learning. ColR, h/s/1909.12200 2019. URL
arxiv. 1909. 1949.
Pierre Jobie, Maxime Haddouche, and Benjamin Guedj. Federated learning with nonvacuous generalisation bounds. orXiv preprint or Kiv::2310..1 11203, 2023.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trendso in Machine Learning, 14(1 2):1 210, 2021.
URL http://dx: org. 10. 1561/2200000083
Robert E. Kass and Adrian E. Raftery, Bayes factors. Journal of the American Statisticol Association, 90 (430):773-795, 1995. URL http://www.jston/ org/staabee/291099
Rahif Kassab and Osvaldo Simeone. Federated generalized Bayesian learning via distributed Stein vari- ational gradient descent. IEEE Transactions Signal Processing, 70:2180- 2022. URL https:
//ieeexplore leee. wtg//ocument////00 9763382
Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar. Adaptive gradient- basedd meta- learning methods. In Advances in Newral Information Processing Systems, volume 32. Curran
Associates, Inc., 2019. URL https://proceedingg www.oddinnnnnnuuis.oooe. cwwrippecccpppprrrrrp httpripss./ppppefffeeeeeeeeeeeee /ttes//ppeee//1///////// flaabebbe2eedae2222211eeeee
pdf
Jakub Konečný, Brendan McMahan, and Daniel Ramage. Federated optimization: Distributed optimization beyond the datacenter. CoRR, abs/15 abs/1511.03375, 2015. URL https://doi..rg//1 /tt.:///0000//1..//////////. 15111
D3575
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression, In Proceedings of the 35th International Conference on Machine Learning, volume 80
of Proceedings of Machine Learning Research, pp. 2796-2804. PMLR, 10-15 Jul 2018. URL https:
//proceedings. mll. www.e/eeessseeeeeeeeeeee srress//80/xuleehov11e. wwweshov18a.hrrr
Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated lcarning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp. 194-797. TEEE, 2020. URL htttps://ieeexploree ieee. orgg//ocuuenn//92115555
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods and future directions. IEEE Signal Processing Magazine, 37(3):50 60, 2020a. URL https://ieeexpllore ieee. ieee. otg//occumet/99845555555
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of FedAvg on non-iid data. International Conference on Learning Representations, 2020b. URL https://doi. org/10.48550/ann (a..50//ariv... 19w..
Qiang Liu. Stein variational gradient descent as gradient flow. In Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017. URL htttps www.:ie...raapapppeeeoss
pdf
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian in- ference algorithm. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL b3ba8f Eahe2f1eeee222222777730333333123 httpr//cttaan.uurp...ccp/pppeeee
16

---

Published in Transactions on Machine Learning Research (02/2025)
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Lactitia Kameni, and Richard Vidal. Feder- ated multi-task learning under mixture of distributions. In Advances in Neural Information Processing Systems, 2021. URL httpps:/ neurips. cwpppper.pppr...peoppppep httpe/cccppppaeeeeeeeeeeee /tless/paper/222///aa 045544094aa00000005444444------0
David A. McAllester. PAC-Bayesian model averaging. In Proceedings of the Twelfth Anmual Conference on Compulational Learning Theory, COLT '99, pp. 164-170, New York, NY, USA, 1999. Association for Computing Machinery. URL httpps://aoi..rgg// 0ttp://0..111//33700033333303330
David A. McAllester. Pac-bayesian stochastic model selection. Machine Learning, 51:5 21, 2003. URL
https://api. http:/pp...aatttt..oooooooorrr// 14704908
Alwyn Van der Merwe and J. Skilling, Mazimum entropy and Bayesian methods: Combridge, England, 1988. Springer Netherlands, 2010.
Darakhshan Mir. Differentially-private learning and information theory. In Proceedings of the 2012 Joint EDBT/ICDT Workshops, EDBT-ICIT '12, pp. 206 210, New York, NY, USA, 2012. Association for Computing Machinery. URL /tttps:///do 21t.:.11111/22226622222228 2320823
Sebastian W. Ober, Carl E. Rasmussen, and Mark van der Wilk. The promises and pitfalls of deep kernel lcarning. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pp. 1206 1216. PMLR, 27-30 Jul 2021. URL https: //proceedings.ct mlr. wttp:///ng........ee///ii/ii./ wwtpppeessseeeeeeeeeeeeeeeeeeeee www.rba..a..tooooooooo..
Anastasia Pentina and Christoph Lampert. A PAC-Bayesian bound for lifelong learning. In Proceedings of the 31st International Conference on Machine Learning, volume 32(2) of Proceedings of Machine Learning Research, pp. 191-999, Bejing, China, 22-24 Jun 2014. PMLR. URL https:// >32/pentinale html
press/
C.E. Rasmussen and C.K.I. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning series. MIT Press, 2005. URL http:///booooggooggll.cccooboeea
Omar Rivasplata, Iija Kuzborskij, Csaba Szepesvari, and John Shawe- Taylor PAC-Bayes analysis be- yond the usual bounds, 2020. URL htpp://////// www.ineuriiccoooo httpa/cpppppeeeleepppppeeeeeeeee paw.es/paper/222///iii. 6399698866a222201144660666600063
http://eeweaeeooooi.o...eeeeeaee pdf
Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: Baycs-optimal meta-
learning with PAC-guarantees. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9116 9126. PMLR, 18 24 Jul 2021. URL attps://proceeding mlr. props///is///occcussslll.....
Alysa Ziying Tan, Han Yu, Li zhen Cui, and Qiang Yang. Towards personalized federated learning.
IEFE Transactions on Newral Networks and Learning Systems, 34:9587-9600 2021. URI. https:
/ / www.smmcccccccolrr..omo hww.aaarrrrrrooppuusee223200000.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference Learning Representations, 2020. URL
https://openrevier wwtp:/neeeee/ddaaannennee
Jie Wen, Zhixia Zhang, Yang Lan, Zhihua Cui, Jianghui Cai, and Wensheng Zhang. A survey on federated learning: challenges and applications. International Journal of Machine Learning and Cybernetics, 14: 2022. URL http::/dx.... doi 10tg:/00.0000//130220222222
Andrew Gordon Wilson Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 370-378, Cadiz, Spain, 09-11 May 2016. PMLR. URL https://proceed::: mlr prr. html
17

---

Published in Transactions on Machine Learning Research (02/2025)
Yuexiang Xie, Zhen Wang, Dawei Gao, Daoyuan Chen, Liuyi Yao, Weirui Kuang, Yaliang Li, Bolin Ding, and Jingren Zhou. FederatedScope: A flexible federated learning platform for heterogeneity. Proceedings of the VLDB Endowment, 16(5):1059 1072, 2023.
Jianqing Zhang, Yang Liu, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Jian Cao. Pfllib: Personalized federated learning algorithm library, 2023.
Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational Bayesian inference. In International Conference on Machine Learning, pp. 06299-263100 PMLR, 2022.
18

---

Published in Transactions on Machine Learning Research (02/2025)
8
Appendix
Reproducibility statement
For all theorems and theoretical results, present detailed assumptions and proofs in Appendix 8.2 More- over, provide comprehensive table containing all parameters utilized throughout the paper in Appendix 8.3.5 This table highlights the interconnections between these parameters and marks the free parameters that can be tuned for optimal utilization of our algorithm.
Regarding the datasets, we employ the FEMNIST dataset, which is curated and maintained by the LEAF project (Caldas al. 2019). We utilize the original train-test split provided with the data, without any additional preprocessing. The PV dataset can be accessed via the following link: https://arive.
com/ ooc.lo/wwnaaaaassSSSilaititttttn
The source code for our PAC-PFI, implementation using GP is accessible within the same Google Drive repository. Upon acceptance, we intend to make the source code for BNN publicly available. To facilitate the use of our software, we have incorporated a demonstration Jupyter Notebook in the source code repository. Furthermore, we have included pre-trained models for PAC-PFL and other baseline models for the PV dataset. Finally, we provide a notebook that generates the figures featured the paper.
8.1 Interpretation through the principle of maximum entropy
In this section, we provide additional justification for the optimal hyper-posterior derived in Corollary 4.4 based on the principle of maximum entropy (Jaynes 1957). The principle of maximum entropy suggests that when only the class of distribution is known, the distribution with the highest entropy should be chosen as the least-informative default. The distribution class can be specified by certain moment constraints, This principle is motivated by two key reasons: first, maximizing entropy minimizos the amount of prior infor- mation embedded in the distribution, allowing for more agnostic representation; second, it aligns with the observation that many physical systems tend to evolve towards configurations of maximal entropy over time. While the principle of maximum entropy is commonly employed to derive prior probability distribu- tions in Bayesian inference (Merwe & Skilling 2010), we utilize it the context of obtaining the optimal hyper-posterior distribution.
Consider the following constrained maximum entropy problem:
max Hp(2) Q
(10a)
s.t. - Ep Ep In
Zz I( m), -E Zs(,,S, Vi (1....,n) .n).
(10b)
In Equation 10a (2) (LL(P) represents Jaynes' entropy with the hyper-prior P serving as the invariant measure [Jaynes 1957). The objective is to maximize this entropy subject to constraints the expectations of In (P,,, S.) under the distribution Q, as given in (TOb). The constants and 7(6i, m,, 6) are determined by Lemma 4.5 and Theorem 3.2 respectively. Below, we establish a connection between the maximum entropy problem and our approach in Section
Proposition 8.1. The minimizer of the server-level upper bound, º", derived in Corollary 444 coincides with the marimizer of the constrained marimum entropy problem presented in [10g)- 110b). when the maxi- mun entropy problem is solved by optimizing the Lagrange function:
arg max (Q( T Σ Ep In Sc) I(ei, m., 8) Ep (P()
(11)
Q
where the Lagrange mulliplier is used for all constraints. The consland. is as per Corollory 44
19

---

Published in Transactions on Machine Learning Research (02/2025)
Proof. Let ²" denote the maximizer of the Lagrange function in are constant with respect to 2, we obtain:
11
By removing the terms in
11 that
e- = arg max (p(2) +
Σ Ep In (P,))..)
(12)
e
According to the definition of Jaynes' entropy,
e- 2
aa arggin min
- KL((PP))
Σ g In (P,
(13)
where we have multiplied the objective in 12 by -1/T and changed maximization into minimization. By substituting the formula for т, one can verity that (13 is equivalent to the server-level upper bound, except for some constant values. Hence, ¹- = Q"
The maximum entropy interpretation of Q allows us to analyze the effect of sampling P from Q on client- level bounds. If client chooses not to participate in FL and decides not to use º", the best alternative approach is sample P from P. The following corollary provides a comparison between sampling P from Q' and sampling P from P.
Corollary 8.2. The i-th constraint in [10b] imposes that the expected upper bound for client is tighter when sampling the prior P from the optimal P.
Q*, compared to sampling from the hyper-prion
Proof. Since a prior which is sampled from the hyper-prior is no longer data-dependent, we utilize the result from Alquier et al. (0066) to derive a bound for client i:
1 (KI(QQ)IP) + BB aa)- + In Inn))) 1
(14)
8m4
which holds with probability at least 1 - over S. ~ D.m... The upper bound in (14) is equal to the upper bound in (4) with S, 0, except for the constant term I. Therefore, the posterior that minimizes the right-hand side of (14) is the same (P.):) derived in Corollary 3.4 with S; = 0. By plugging (*,P,S.)
into [14h, we obtain the counterpart of (6):
8b (b--a)²
a)² + )
())(()))) S In (P,
(15)
8mn;
holding with probability at least 1 over S, D The i-th constraint in (00b) is obtained by taking the expectation of the upper bounds in and 15 when P Q* and P ~ P, respectively, and enforcing that the former is smaller than the latter.
According to Corollary 8.2 However, since we solved the constrained problem per [0ap (10b] using the Lagrange method with a single multiplier, the constraints might be violated. The constraints are more likely to be satisfied when 7 is large, which can be achieved by having small n, n2, and B, while simultancously having a large value for Ag. other words, when there are fewer clients participating FL, the hyper-posterior is more likely to provide improvements for those clients.
participating in FL is beneficial for client i if the i-th constraint is satisfied.
Remark 8.3. In a similar manner, we can avoid any PAC arguments and use the principle of minimum cross entropy for calculating the posterior. With these two rules, the optimal posterior and hyper-posterior are the same as those obtained by minimizing client-level and server-level PAC bounds.
8.2 Proofs and derivations
We first mention without proof powerful lemma, called the change of mcasure incquality, that the basis of proving PAC bounds in most papers. The statement below is adapted from the Appendix of Pentina & Lampert (2014).
20

---

Published in Transactions on Machine Learning Research (02/2025)
Let f be random variable taking values in set A and let XI,..., Xt be independent random varäables with each X distribuled according to Ps over the set Ag- For functions A x Aş + R, k = 1, ,1, let Eu(f) = Ex go ( Xx) denote the expectation of g under X5 ~ Pk as a function of f. Then, for any fized distributions n, over A and any y > O, we have that
Lemma 8.4 ((Pentina & Lampert 2014
Eynp Σ (Ex(I) - gh (x((, & - kL())T>
1 (17))
where (07) = In Byna e² Ei
(Su()) (
is referred to as the log noment-ggneratie function.
When &a - 9k is bounded, we bound the expectation of the log moment-generating function in Corollary 8.6 which uses the Hoeffding's 's lemma stated below.
Lemma 8.5 ((Hoeflding) 1963)). Let Y be a zero-mean real-valued random variable such that E [a, almost surely, t.e. with probability one. Then for any 7 > 0:
Bie S (0-
Corollary 8.6. 1f (J) (u, (f, [ax, bu] almost surely for all f and X it holds for every that Exieps" ... Ex. e v(1) S ã Ek_1(b--a
Proof of Corollary 8.6 By taking the expectation of the moment- gennenttgeneating function w.r.t every Xk.
E Exir Ex (1) e
Eximaa
Ex Eswn
II
((L())
A=1
II e (5(00)
= =Esaa ... Exrry
where in the last line we have changed the order of expectations. For given f, the terms Ex(f) gu (5 Xx) for & (1,..., are independent from each other which allows applying Lemma 8.5
Exysm Ex,,n, 2 ESnn
II
x Nam 7 (celf) ((alf)
II km1
( -aw
22 El-,0
(be (bk-
$ Exor
e
Since 1/7 S 7, we can use Jensen's inequality
Jensen 1906 to write:
o 1 x() < El_, (be (be-s))
E Ex Ex
One can utilize Markov's inequality remove the expectation Corollary 8.6 and obtain a probabilistic bound on (7)). We will use Corollary 8.6 multiple times, and thus, postpone applying Markov's inequality to avoid simultancous stochastic incqualities which must be combined with union bound argument.
8.2.1 Proof of Theorem 4.2
The proof of Theorem 4.2 is carried out in three steps. The first two steps bound the true risk of clients with and clients with enlarged datasets, respectively. By merging these two, we will obtain a bound
According Markov's inequality, if X is nonnegative random variable > O, then Pr[X 2 al S E[X)/a.
21

---

Published in Transactions on Machine Learning Research (02/2025)
In the last step, the closed-form formula of the optimal posterior is exploited to make some simplifications. The main assumptions are independence of clients (guaranteeing independence of X5 in Lemma 8.4p, boundedness of the loss function (needed to apply Corollary 8.6), adoption of the optimal posterior (yiclding the bound per ( and existence of a finite e E R2, such that sampling P from @ preserves e-DP for all clients (required to use Corollary 3.4
the server-level true risk.
Step 1. We apply Lemina 8.4 with the following instances: take / - Σ m, and assign random variable
to each observed sample by clients, X₁ Zij. where Zij is the j'th sample of S; Let a : {1, ... .I3 + {I,-.-, be a mapping from each random variable X to the corresponding client, a(k) = i if X = Zij for some j. Correspondingly, we take Akk = Da(k) to be the respective distribution. Further, we set f (P.h..... ,h,) to be tuple of one prior and n hypotheses and consider distributions æ == (P, P...... P) and p -= (2, (P, Q* Sn)) over it. Each function gk is designated to be one of the summands in the empirical server-level risk, 9u (f. Xx)
-_((h (han())) Xx). By invoking Lemma 8.4 with y A 2 1, we mma())
have:
n E Ep~e Σ ( s
"
Ep E e Σ
L (Q)())))
(*)
n
n
1 KL
(L(OIP)) + Bp=oKL((((P.S))1)
(t)
1
1 (A1).
(16)
A
Line f is equal to KL due to (13-15) in Rothfuss et al. 2021 and (A1) is a log noment-generating function defined as:
w (A1) = In EppeEE.EEEe Ei (BawD, ((h,2) -, Exes, ((h,z))
For E ba, we apply Corollary 8.6 acknowledging that 15x 9kl S (b a)/nma(())) hence, obtaining:
Bs ES..D ex (As))
< C A1 (b-a)²
(17)
The right-hand side of from the true risk in as new clients and new samples are missing.
matches the server-level empirical loss per
S but the left-hand side is different
Step 2. In the second step, we 1 = n and assign one random variable X5 to each client. To maintain cohesive notation,
will use subscript instead of & for elements of Lemma 8.4 We substitute each X; with subset of m, samples from S; drawn without replacement, which is possible
my S mi. Set f f P. P, p= = Q, and Xx) = c (Q, D.). Notice that S, and D, are embedded in the definition of ga and not given as function arguments. Let X2 = A/(n2 + v) > 1, where n2 is the number of clients with m, > 0 and u is a small positive number. By applying Lemma 8. with parameter 7 = X2, we obtain:
E P 2
B
(Q( S S- Ep_o Σ
c (Q'(P,
(0)
KL(2))) +
b2 (2)))
(18)
A2
Σn im (F (oaDp cf ( (P.S....)) - - (o....))))
{2(22) EnE Epppe
When m for all clients, the two sides of tor are equal, 2 ()2( is zero, and the weight of the KL term in 18 goes to zero v O. Thus, by setting d2 proportional to na v, 166 and (8) are consistent.
22

---

Published in Transactions on Machine Learning Research (02/2025)
It is evident from the definition of & and 9: that K5, 9al I (6 -)-aa/n. Still, we can exploit the explicit formula of Q" per 5 to obtain a potentiilly smaller range by controlling the effect of a new sample of size m, on the optimal posterior. Let Q = (*,P, S,) and Q = Q" (P.S, U S.) for some fixed P and S, From
the closed form formula of Q² per 5). for every h E H:
(h)
E "ii E EnEsIu8, ((1.))
- En pe
xES, ((h,z)
(19)
(h)
2ocs, ((h.z)
En my ti ZRES, ((b,z)
From the bounded loss assumption, we have:
Σ ((),z) z)) S m4 a)
E(h,
(20)
mi + mh
mi
mi + m
ZE S.S.
xES:
From 11) and (20),
(h)
[e matmm
200
(b-o) a)
mh m (b- o0 which obtains bound on E:(f) xi);
(1(h)
(U D.) - (P.))))) D.)
1 E D.
f.ff) ( (f)
n
1
E Dm Ez-D, ((h, (Q) Q7
7
E
C (Q)" Si), Di). m
(b-a) 1, e
28m; (b-a) -1] 1
b
-000 (b-a) - 1): (e " mitmis
2010 (b-a) -
(21)
Comparing 21 and the naive incquality 15: - gi S (b- - a) /n, it can be verified that 16: - 9sl S A., where:
A min (b b(e m.m (b-a)
23m;
-20m mi (b (b-a))).
It is possible to obtain tighter range, A; (b a)/n, when my is large, my is small, or B is small. Particularly, for m = 0, A, = 0 but b - provides vacuous bound.
By applying Corollary 8.6 one obtains:
- Es, D. (A222(A2) <e A/ Σ A3 S Sesn
Ag (b-o)?
(22)
Note that S # Ø required for defining X4. Hence, new clients are not added the analysis yet.
Merging steps 1 and 2. Bringing 16 and 18 together, we gel:
(S,,,,
( +
KL(Q)P)
+
1 Σ Bp-oKL(Q'(PS)))))) A
(1)) + A1
Aa (22))
(23)
provides a bound on the server-level true loss and incorporates various components such as the empirical loss, complexity penalty terms (represented by KL divergences between the posterior and the prior, and between the hyper-posterior and the hyper-prior), and two log moment-generating functions.
Equation 23
Next, we bound the weighted sum of the log moment semnentteeneratin functions in 23 when E E [a, One obtains from Markov's inequality that:
Pr (( (2(2)) <
1 11(bma) an2
En iim m. 2/ Σ. Af
g
R D. ... ES,,DD SINDD ... R. Dini
exi vi (Ar)+ 22(22)
1
(24)
A110-a1²
D"
1 Aa
A3
b E 8n2
my
2
23

---

Published in Transactions on Machine Learning Research (02/2025)
where the probability is taken over S; " and S; ~ Di for i = 1,... ,n and d e (0,1) is the confidence
level given in Theorem 4.2 To bound the right-hand side, we multiply 177 and [22h, which is possible as they involve expectations over independent random variables, resulting in:
~ "S, D ... E e (all v2 vz()2)
A1
E +
E oi
DS ...
Sc
(25)
By putting together 24 and
25 we derive:
Pr 2i (A1)
1 e22 S
A1 a)²
A2
Σ
n
A + In
22
8n²
m
8
Pr e (n x/2 (2(22)
1
A (bmma)
on
Σ
iim
+
Em=
4/
2 5,
(26)
0
where the probability is taken over S D. and S; D for i 1,...,,
One obtains from
23 and
26 that:
Pr Pr( s
( + + +
1 )
+
1 a
Σ
E Ep Bp..K((Q(PP.))11P))
X
A1 aa)
+ de
> - +m
2 2
(27)
8n² m
Step 3. In the final step, we utilize the closed-form expression of the optimal posterior for each client, Q", based on Corollary 3.4 to simplify (27). By substituting the definition of the server-level empirical loss given in S into 27 and refactoring some terms, one oblains:
Pr (2,,,, SEP~O
co (P))))) + T -K(((((.))))))) Xi
(+)
1/- )
A (b a)²
M
A +Im
1 2--
21
8n²
When we set \1 n.8, each term in the summation in
! becomes equivalent to the upper bound for client
in Theorem 3.2 up to a constant. This choice of ê enables us to simplify f similar to 6 resulting in:
Pr
Ep Ep-o
2
(P.S.)
1
+ {L(()))))
nB
nB
A2
8(b
a)²
Na 8
Σ
In
{ 21 1-8.
8n
m
Note that the selection of A = n2 is only feasible when B 2 1/n because the first step of this proof requires A 2 1. This is the reason behind the assumption 3 > 1/n in Theorem 4.2 By substituting X2 >/ +)). which was motivated in the second step, get the desired bound. Finally, the factorization proposed in Rothfuss et al. 2021 to convert In(1/5) into In(1/6)/vn.
8.2.2 Proof of Lemma 43
Assume m, is only known when E A, where AC (1.... .n...n)) n. is a subset of clients, but unknown for the rest of the clients, B (1, ......nn/o... A.. In an extreme case, Ø means that m, is unknown for all clients. Let p(A) and pBB) denote the number of clients with m, > in sets A and B respectively. It is
24

---

Published in Transactions on Machine Learning Research (02/2025)
clear from the definition that p(4) p(B) n2, where p(A) is known but p(B) and ng are unknown. Also, p(4) $ |A] and () < [ where |4] and [B] stand for the cardinality of the respective set.
We prove that the upper bound in Theorem 4.2 is looser when replacing A with (6 a)/n for E B and n2 with p(4) + [B]. is enough to show:
P(A) (B) +U KL(Q)P) + ΣiEA + iEB A 8(((A) + (
A KL(Q)P) + Σies 4 + IB( b ((((A) + B A.
À (AA + B +U U
Since p(B) < [B] and KL divergence is non-negative, we will prove that:
EiEn A Eien EiEa A + BB 2))) p(A) pB))
(28)
p(A) +U
For clients with m, 0, A, 0, and for the rest, A S (b - a)/n. (b-a)/n- and a stronger condition than is:
Hence, Σien S ((b-(nn))))
EiEA A + p (B)
EiEA A
Σ A s (o(A)+ + +2) 00)
((A) + pl +
P(A) + B +v
71
EEA
which holds since Σie A ((A)((b- a)/n)² and v 220 0.
8.2.3 Proof of Corollary 4.4
The optimal hyper-posterion posterior minimizes the upper bound on the true server-level risk established in lizing the structural similarity between the server-level and client-level upper-bounds, equations 9 and we employ Corollary 3.4 to obtain º*
9 By uti-
8.2.4 Proof of Lemma 4.6
We invoke Lemma 8.4 4 by defining X, (D,, St) and using X; = [x to distinguish between components
X, The distribution over X, is = (T, (ml) Additionally, I - n, / P. π P. p = º*, and gi (f = a cC (f. X; []), (DI) are used. While gi defined above resembles the one from the second step of Proposition 8.2.1 the difference lies in considering S, and D; as part of the random variable X,, giving rise their presence in the expectation. By applying Lemma 8.4 with
y Á 2 1, one obtains:
ECD. Bp- E n
Σ c (Q))8)))
kL(
1 (())
(29)
The left-hand side of 29) is the true loss for generic new client, i, sampled from T. The moment generating function and the bound on its expectation due to Corollary 8.6 are:
(BBo.mJArTB e (P.)))))))))). e (1(P.)))))))
(A) = In EpapE
E(D ... EE E.D. ... Es.. e (()) < t (0-0)²
(30)
25

---

Published in Transactions on Machine Learning Research (02/2025)
Merging (6) and the result of Step in the proof of Proposition 8.2.1 when \ 1/(n3) and Q" is the
optimal posterior mapping in. 5 we rewrite 29
as:
BD.TTBS,DDT" B.p..p...p.rrr..... L L9 (.)))
E P In Z3
n5
1=1
KL((T)P) + 1 = nB vn(3)).
nB
Plugging in the formula of Q' given in Corollary
4.44
and setting \ = a/(n2 r)),
1
+
ED.TTBs,wyy ( S
n2 +
In 2 (P.S)
n2 + A
11 (n
+
n
+ v
Using a similar technique to the proof of Proposition 8.2.1 along with result.
17
and
30
we obtain the desired
8.2.5
Looser upper bound for new clients
We consider Theorem 4.2 when Q = Q" and simplify the terms in the upper bound that involve Q*:
1
n2 + +U
B In (P. +
+
(L(2)1P)
n
n3
A
1
n o ( (-T)
(P,S.) +1n 2'(P) P(P)
(31)
i=1
1 n,
In
+
In (P,
(32)
nBT
Equation 31 follows from the KI. divergence definition and 32 obtained by substituting the closed-form
formula of dcrived in Corollary 4.4 into the expression. To compare the bound in Theorem 4.2 with
Lemma G we can substitute the first two terms in Theorem 4.2 with (32). In order for the bound in
Lemma 4.6 to be looser than this substituted bound, it is sufficient to show that Eis A3 < (b - a)2/n.
which is truc based on the definition of A; in Theorem 4.2
8.3 Details of the algorithm
8.3.1 Asymptotic behavior and non-vacuousness of bounds
We analyze the asymptotic behavior of the client-level and server-level bounds the number of samples per client and the number of existing clients approach infinity, i.c., m, oo and co. A PAC bound is considered consistent when the gap between the true and empirical risks goes to zero. The client-level and server-level bounds are consistent if and only if: a) B E ((1), b) 8 E o(m;), c) X E ((zz), d) À E o(n(n2 + +)). Here, o and Ω represent the small-oh and big-omega notations for function growth rate (Cormen et al. 2022
Non-vacuous bounds are essential, ensuring that the terms independent of the posterior or the hyper-posterior are not excessively large such that the bound on the true loss holds regardless of the empirical loss. To achieve non-vacuous bound, is necessary (but not sulficient) that b- condition can be converted into an upper bound on based on the results from Lemma 4.5
and that e; < /2(b - a). The latter
In our experiments, we follow Rothfuss et al. (2021) and set 3 = m, + m,, leading to non-vanishing gap between the true and empirical risks al the client and server levels. However, this choice simplifies the computations as discussed in Section (5), Moreover, this choice leads to faster decay the KL term in both the client-level and server-level bounds, which can be advantagcous when m, small (Rothfuss et al, 2021) We tune X, while respecting the non-vacuousness condition, manipulate the regularization strength of the hyper-posterior towards the hyper-prior.
26

---

Published in Transactions on Machine Learning Research (02/2025)
8.3.2 Background on GP
Below, we provide additional details regarding the GP models we use. For more comprchensive overview of GPs, please refer to Rasmussen & Williams (2005). In the rest of this section, we express the dataset of client i as S; = yi), where X; E Rx C ia is a matrix with each data sample as a row. The corresponding target values are stored in the vector y, E Ry C R
ka) denote GP prior specified by a deep mean function, Ma a deep kernel function, kip and a Gaussian likelihood with noise standard deviation, OΦ E R4. The vector Φ R concatenates all learnable paper-pprameters of the GP prior, including the mean parameters, kernel parameters, and the likelihood noise. The mean function, m : Rx + R, implemented as multi-layer NN with weights given by Φ, hyperbolic tangent activation functions in the hidden layers, and linear output functions in the output layer. The kernel function is suuared-exponenteee (SE) kernel applied on top of an NN, defined as:
GP with deep mean and a deep kernel (Wilson et al. 2016). Let
(x(xxx)) = exp( [1f(()) (x)) [0. vx Rxi Vx E Rx.
The function fø Rx + Rdd represents an NN with weights given by & that maps the typically high- dimensional feature vector, x Rx C Rd, to a lower-dimensional output vector Rer, where our specific case, dy two. Deep kernels serve as feature extractors and allow for learning more sophisticated representations from the data (Ober et al.] 2021). The length scale of the SE. kernel is set to because the weights of the output layer of fo can be freely chosen to compensate for it.
Computing the LML. As mentioned in Section 5 when client employs a GP prior, Pø. the negative log likelihood loss, and sets my, the quantity In (P (p,Ss) S.) defined in Corollary 3.4 corresponds to the LML of the GP. The closed-form formula for the LML is as follows:
In (P 8:) = In Pr/y Pal (y: mp.i)
(Kp,i + to - -y;
mş.))
In K + o 2
mi In(2x),
(33)
2
where ||| denotes the determinant of matrix. The vector m Rm contains the output of ms applied each data sample and the matrix K Kg. pm; is the kernel matrix associated with the kernel function K applied to X,
The delerminant term in 33 is commonly viewed as a complexity penalty for the kernel (Rasmussen & Williams 2005). However, recent findings in Rothfuss et al. (2021) suggest that this form of complexity reg- ularization may be inadequate when dealing with expressive kernels that possess numerous hyperparameters, such as deep kernels. Furthermore, there is no complexity penalty imposed on the prior mean, which can lead to higher risk of overfitting. In PAC-PFI, tion, which includes KL divergence term between the hyper-posterior and the hyper-prior. The KL term penalizes hyper-posteriors that deviate significantly from the hyper-prior, thereby elfectively regularizing both the mean and the kernel of the GP prior.
address these limitations by using (g) as the loss func-
Computing the predictive posterior. Given set of priors Pæ.. Por trained by PAC-PFL, our objective to compute the predictive postcrior a test point x,, We denote by Q uniform distribution over these priors, which serves as the SVGD approximation the true optimal introducing a categorical random variable Z that is distributed uniformly 1,... k and denotes which
º.. By
27

---

Published in Transactions on Machine Learning Research (02/2025)
of the priors Pei.... Pos is used for making inference, the predictive posterior be expressed as follows:
Pr(y./x,, Si, ②
Σ Pr(y.|x, x. S, = x Pr|Z
x| x/x., ②
Priy.[x,,Sh. P/. Pr|z = k|x,,, Si, 2
Pr|y- St, Pa.. Pr[xx, S: Pdn Priz = K|
(Bayes rule)
fr[x,, S
x=1
1/k
Σ Pr|y, (0, S₁ Pæ.. Pr[x Si|P Pr[x., S [
(Ô uniform).
(34)
The first term in the summand of (34 is the predictive posterior distribution for client corresponding
the GP prior Pæ. (h) GP(h)mp ks.), which is given by Rasmussen & Williams (2005)
Pr|v.|x., Si, Pa.. (y En,,
(35a)
Mx == m.r (X.) + k2 Kerii --- bunt + to () (y)
(y: mond),
(35b)
Σx == kp. (X., X.)
kT pn (K + 02 1) - "kp... + o I.
(35c)
where kg.i,= R is computed using the kernel function K& between X, and x..
Based on the assumption that x, is independent of all samples in S,, we can expand the second term in as follows:
34
Pr[x., S. Pr|x,| Prrp P Po.l
-rr[x.] Xx Pr,y, P
= =Pr[x.] Xr[X.] W (W m.. (X.), Kg. (X,, X,) + 02 I)
(36)
where the last line is the predictive distribution of GP before conditioning on the observed data.
By merging
34
36
one obtains
Pr|y.Ix., St, Ó
Σ On No NyooMee En);
(37a)
Pr[x.] Pr[X
N (y:/mo, (X:), ko. (X,, + o [0,1]
(37b)
1: Prr(x.: S
It is straightforward to show that Σx= an 1 in equation 37b) As result, the predictive posterior in equation (3aa is valid distribution over y..
8.3.3 Background on BNN
In this section, we introduce Bayesian Neural Networks (BNNs) used for classification tasks. Let he Rx Ry represent neural network, with 0 E O denoting its parameters. Utilizing this mapping, we establish the conditional distribution as Categorical distribution, derived as follows:
Pr y n, e = Pr[p] Categorical
(softrmaac (softman (hotmar(hoox)))))
Computing the LML. Unlike for GPs, the LMT,
In 1o (Po, St) In Eo m Exes, 6(ha.2)
28

---

Published in Transactions on Machine Learning Research (02/2025)
is intractable for BNNs. Instead, use the following formula, proposed by Rothfuss al. 2021 to approximate the LMI:
In (P S2) = In LSE - Lnee,
= (e Exeo, e(he, (2)) Ii L,
(38)
where LSE is the LogSumExp function. This formula involves drawing I. samples 0,,... 0r approximate the LML. We utilize this formula for LML approximation in the context of BNNs.
from PΦ to
8.3.4 Background on SVGD
SVGD approximates a target probability distribution using discrete uniform distribution over a set of desig- nated samples called particles (Liu & Wang. 2016). Through an iterative process, the particles are updated by minimizing the KL divergence between the estimated and the target distributions in the reproducing kernel Hilbert space corresponding to a kernel function, kSvGD- We utilize an RBF (Radial Basis Function) kernel with a heuristically chosen length scale, as described in Liu & Wang (2016). In our framework, we aim to obtain samples ....... ... P... from the target distribution Q'. To simplify the notation, we represent the particles as $1.-.. ; ΦR, where each particle Φb. fully characterizes the corresponding prior, Po.. Accordingly, we write P(),) and (Φxx) instead of PPP..) and Q* (Pa.).
Initially, the particles are sampled independently from the hyper-prior. Then, at each iteration, each particle Øx for {1.... }} is updated according to the following rule:
Σ (ksvap(0). Φx) Vo. In (OI) + Vo.ksval(e, ow)), Φx t Ø k
(39)
where n) is the learning rate the current iteration. By utilizing the formula for Q" provided in Corollary 4.4 when mi, can derive the following expression for E f1,... ,k}:
V In (xx) Vo. In Pn)()))) +
Ven In 2 (Po..
(40)
where r is defined in Corollary 1.4 By comparing 33 and 140 it becomes evident that the data of client is solely involved in the SVGD update through the term V Φa In 7C (Po., S;) for each particle Φx.- As a result, the clients only need to transmit the gradient vector, V In 2 S.).. V In (Pa. S,) or an approximation of it using a mini-batch approach, to the server. This observation is the intuition behind the sub-routine Client Update in Algorithm which we present in detail in the next subsection.
8.3.5 Table of parameters
We provide a comprehensive overview of the parameters relevant to our theoretical results and to our algo- rithm, along with their interrelationships in Table | To enhance clarity, the parameters are categorized into three groups, separated by horizontal lines. The first group describes the fundamental attributes of the FL problem, such as the number of clients, and is set externally. In the second group, we list the parameters central to our theoretical findings, elucidating their relations to other parameters. Lastly, the third group encapsulates the paramcters used in implementing Algorithm D explaining their role in the practical appli- cation of our proposed hothoolgg Importantly, it should be noted that the constraints on the parameters have been deliberately set to ensure non-vacuor bounds, as discussed in Appendix $.3.1
8.3.6 Computational complexity
In this section, we discuss the computational complexity of a single iteration in training PAC-PFL, specificaly when employing the log-likelihood loss and B = mi as outlined in the paper. At the client level, the computational complexity hinges on the calculation of the LML. For GPs, this complexity is Okm))) where i is the number of SVGD particles, and m, denotes the size of the training datasct for the specific client i. For BNNs, the complexity is OkLmm), where L corresponds to the number of samples used to approximate the LML, explained in Appendix 8.3.3
29

---

Published in Transactions on Machine Learning Research (02/2025)
Parameter
Description
Domain
Selection experiments
number of clients
E N
given in the dataset E {0, ,n]
722
nunber of clients with new samples
problem setup
T
distribution of clients
unknown
m
number of samples for client
E N
given in the dataset unknown
D;
data generating distribution of client
(Di, mz) ~ T
S;
dataset of client
Dit
given the dataset
m;
number of new samples for client
E {0.... ,
given in the dataset given in the datasct GP and BNN
S;
new dataset of client i
D.
H
model class
E aa b]
client inference
loss
negative log-likelihood
s.t. - a < 8
P
prior
over
~
Q
optimal posterior for client
over H.
Corollary 3.4
temperature of Q;
E IR+
m; (Appendix 8.3.1
8
confidence level
E (0,
code
4
DP parameter
E I.+,
28r(b-a) (Lemma 4.5) m,
P
hyper-prior
Gaussian (0, of I)
server inference
00
variance of P optimal hyper-posterior
E R4
tuned by vross-validation
Q"
Corollary 4.4 10-4
constant in Theorem 4.2
E R.
2 n2 +
constant in Theorem 4.2
tuned by cross-validation
s.t. 4 S 22 a)
temperature of Q*
E IR
X+ (Corollary 4.4
learning rate number of SVGD particles
E IR+
tuned by cross-validation
algorithm
E N
see code
number of iterations
E N
see code
number of clients per iteration
E [1,
m]
see code
b
data batch size for each client.
E {1,
(I, mm
code
number of samples for estimating the MLL of BNNs
L
E N
code
Table 4: Summary of the employed notation and their relations. The parameter domains are set such that the bounds non-vacuous. The parameters categorized into four groups separated by horizontal lines: the first group describes the characteristics of the PFL problem. The second and third groups enumerate the notation used in our theoretical results. The final group encompasses the parameters in Algorithm 1
At the server level, the computational complexity varies based the type of hyper-prior used. When utilizing a hyper-prior with diagonal covariance matrix, the complexity is O(ck + k²),, where represents the number of clients selected per iteration. a hyper-prior with a full covariance matrix is used, the complexity increases to O(ck + 1.3).
The training time (in seconds) of PAC-PFL and some baselines (where we optimize the computation time as much as possible without sacrificing the performance) for our classification experiments are provided in Table 5 As can be seen in the table, while the training time of PAC-PFL exceeds that of the simpler baselines, it remains within the same order of magnitude.
8.4 Role of DP
We adopt DP in two key ways. Firstly, we utilize DP ensure that when the server samples a prior distribution from the hyper-posterior and releases it, this published prior distribution does not raise privacy
30

---

Published in Transactions on Machine Learning Research (02/2025)
ALGORITHM EMNIST
FEMNIST (20)
FEMNIST (500)
FedAvg
900
120
480
MTL
950
120
500
MAML
1100
540
605
PAC-PFI.
1260
960
970
Table 5: Training time (in seconds) of PAC-PFL and some baselines. The training time of PAC-PFL is slightly longer than that of more basic baselines but remains within the same order of magnitude.
concerns for the existing clients. This application is akin to typical scenarios in FL research where DP is used to protect the model shared by the server from revealing information about clients' data. Besides addressing privacy concerns, we use DP to establish a PAC bound at the client level. As illustrated in Figure [ our setup involves prior distributions that depend on the data existing clients. While most PAC bounds assume the prior distribution was selected before observing any data, we leverage results from Dziugaite Roy (2018) to derive a PAC bound that holds when data-dependent prior is obtained through differentially private algorithm. To the best of our knowledge, DP has not been employed for this purpose in previous FL methods, Below, discuss the role of DP in our ideal setup (Section 4 and our practical algorithm (Section
Inherent privacy of the optimal prperrposteerir In Section 4 we considered a specific family of hyper- posteriors, where sampling a prior from the hyper-posterie satisfies e-PP for a finite e. This assumption simplifics computations and enables us to use the closod-form postcrior provided in Corollary 3.4 We then established the server-level upper bound in Theorem 4.2 for hyper-posteriors meeting the privacy criterion. Interestingly, the resulting upper bound does not contain the parameter This can be intuitively explained by the fact that the hyper-prior is chosen independently of the data, making the server-level scheme akin to typical PAC-Bayesian bounds that employ lata-independent priors. Additional details can be found in the proof presented in Appendix 8.2 In summary, the c-DP assumption facilitates the computations by allowing us to use the optimal posterior formula without directly impacting the server-level bound.
By minimizing the server-level upper bound, we derived the closed-form formula for the optimal hyper- posterior in Corollary 4.4 Subsequently, we ensured that the optimal hyper-posterior satisfies the privacy assumption started with. To achieve this, Lemma. 4.5 determined the value of for which sampling the prior from the optimal hyper-posterior meets e-DP. As this e is finite, it confirms that our optimal hyper- posterior belongs to the family of posterposteriors we initially considered, thus completing the derivations. While assuming a finite c is sufficient for deriving the intended results, it is worth noting that the client-level bound becomes looser for larger values of e. Further discussion on providing non-vacuous bounds can be found in Appendix [8..1
In the idcal setup, DP arises from the inherent randomness in sampling from the optimal hyper-posterior. As result, DP is achieved without the need for externally injecting noise, which is a common practice in typical DP mechanisms. We refer to this property as the inherent DP of the optimal hyper-posterior.
Loss of inherent DP due to SVGD As discussed in Section since SVGD a deterministic sampling algorithm, we forfeit the inherent privacy of @ Consequently, the client-level and server-level bounds no longer hold for the approximate syper-posterion Instead, we rely on the premise that if SVGD effectively approximates the optimal hyper-posterior, the empirical and true risks of the approximated and optimal hyper-posteriors should be closely aligned, suggesting the bound's validity.
Differentially private PAC-PFL To reintroduce e-DP after SVGD approximation, Algorithm 1 can be modified, drawing inspiration from differentially private FedAvg (Geyer et al.) 2017). In this modified version, the server clips the norm of the gradient sent by cach client and introduces noise to the aggregated gradient. For simplicity, we will illustrate the algorithm using only one SVGD particle. A pseudo-code for this private version of PAC-PFI. is provided in Algorithm g
31

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm 3 Diflerentially private PAC-PEL with 1 SVGD particle executed by the server
Input: privacy parameter e, gradient clipping norm 7, hyper-prior P, parameter 7, number of iterations T, number of clients per iteration c, mini-batch size b, learning rate
Initialize prior PΦ ~
Initialize
3: for 1= 1 to 7 do
4: Select a random subset C, of с clients
for each selected client in C. in parallel do
G; Client Updale(b, Φ)
Clip gradient norm: G; +- G:/ max [
7:
Collect clipped client updotes
Sample noise: v Lap(O, Tv Idp)
G - Eiec. +i+
Aggregate gradients and inject noise
10:
for A - to k do
VΦ. In (' t V In P(ox) + GT.
Gxa is the K-th row of G
11:
12:
for == 1 to k do
Øx + Φx + 1 E (ksvcp(0),0)) In ( + Vo,ksvau( Φw))
13:
SVGD update
14:
return Por
Differentiallyy-rre privale SVGD apprerimation of Q"
As considering single particle in Algorithm 3 the Client Update function returns vector g instead of matrix G, as is the case when using multiple SVGD particles. In Algorithm 3 the notation T7 Lap(O, ca IdØ) represents a multivariate Laplace probability distribution. In this context, 0 denotes zero mean vector, and Tt IdΦ specifies the scale parameter applied across all dimensions. Both the particle, o, and the noise vector, v, have dimension dø.
<c
The noise scale in Algorithm 3 grows linearly with the number of iterations, T. This can introduce significant noise into the gradients, potentially impairing both convergence and the overall accuracy of the algorithm. This challenge is intrinsic to differentially private gradient descent and not unique to our model or FL in general (Bagdasaryan al. 2019). Investigating alternative strategies to achieve e-PP remains promising avenue for future research.
We assess the performance of Algorithm using the Polynomial dataset, introduced in Appendix 8.5 Specif- ically, we use 120 existing clients, each with 10 samples. We vary the privacy parameter and examine its impact on the algorithm's performance, measured by the average RSMSE metric. Figure illustrates the RSMSE across different values of e. Additionally, we plot the RSMSE of the non-private algorithm, Algo- rithm D in red for reference. As anticipated, the model's performance improves as increases, signifying a lower level of privacy and consoquently reduced noise requirements.
8.5 Datasets
8.5.1
PV dataset
The transition renewable energy sources, such as roof-top photovoltaic (PV) panels, is crucial to address energy challenges; however, the intermittency of solar energy remains a challenge that requires accurate prediction of solar panel power output. Predicting the time series of PV panel power outputs requires either specific measurements large amounts of data. A potential solution is to design collaborative methodology among multiple PV datascts collected at nearby locations for predictive modeling.
We work with an hourly simulated dataset of PV generation time series from rooftop panels in Lausanne, Switzerland. Our objective is to predict the next-hour PV generation, utilizing 15 features encompassing auto-regressors and weather data. We investigate four scenarios: PV-S (150), PV-S (610), PV-EW (150), and PV-EW (610). In the PV-S variants, all houses face south, while the PV-EW variants involve houses oriented either east or west. The PV-EW scenarios, being bimodal and highly heterogeneous, present
32

---

Published in Transactions on Machine Learning Research (02/202))
1.4
not private
1.2
RSMSE
1.0
0.8
0.6
0
2
4
6
8
10
epsilon
Figure 4: Performance evaluation of differentially private PAC-PFI, CCC-PFF.(AAoooo (Algorithm) 3) on the Polynomial dataset introduced in Appendix 8.5] The RSMSE metric is plotted against varying values for the differential privacy parameter, Lower values correspond to lower privacy levels. The RSMSE of non-private PAC-PFL (Algorithm D is shown in red for reference. As expected, performance improves the privacy level decreases due to lower noise injection.
greater modeling challenges. To assess the impact of dataset sizes (denoted m), consider two scttings: clients with either 150 or 610 training samples, corresponding to two-week and two-month data windows, respectively. The number in each dataset's name indicates the training sample size.
Data generation. To generate simulated datasets for the PV-S and PV-EW experiments, we first obtained a real dataset of hourly solar radiation and meteorological measurements in Lausanne from the Photovoltaic Geographical Information System (PVGIS) online tool (PVG). We then used the puláb Python library (Holm- gren et al. 2022) to simulate the PV power output based these measurements. To simulate the PV panels in different houses, we sampled meteorological data and installation specifications from normal distributions with specified means and standard deviations. For example, in the PV-EW experiment, the azimuths of the PV panels were sampled from a bimodal normal distribution of (N 15%) + N2700, 15")), where M denotes the normal distribution. Figure iilusttrates the power output profiles of 24 houses in the PV-EW experiment over five days. This figure demonstrates that the curves have noticeable differences; however, they show similar trends. As expected, the houses facing east or west are more similar to one another, while the differences among different sub-populations are higher. For instance, the peak production of the houses facing the east occurs in the morning, whereas the peak for those facing the west occurs in the afternoon.
To ensure that each client's 's dataset is identically distributed, despite the variations in meteorological patterns throughout the year, we filter the dataset down to the months of June and July. We employ two distinct training datasets for our analysis. The first dataset comprises the initial two weeks June 2018, which provides a total of 150 samples for each client. The second dataset encompasses the data from both June and July 2018, resulting in 610 training samples per client. For all experiments, the test dataset consists of the data from June and July 2019. Furthermore, we exclude nighttime data points when the PV generation is zero. To normalize the data, we standardize each house's features and output, setting the mean to zero and the slandard deviation to one.
Features. The dataset contains various meteorological features, including solar beam and diffuse irra- diances, temperature, wind speed, solar altitude, lime, and dale. Solar beam and diffuse irradiances are particularly valuable in predicting PV generation, but their measurement can be costly. To reflect a realistic sconario, we assume that all houscholds have access to the irradiance data recorded at a specific location in the city, such as weather station. However, they are unwwae of the specific irradiance values at their houses, which may differ from the weather station. The temperature and wind speed may vary slightly
33

---

Published in Transactions on Machine Learning Research (02/2025)
175
150
125
power
100
75
50
25
8
12
16
10
14
8
13
17
11
15
13
hour of the day
Figure 5: Power output profile of 24 houses the PV-EW experiment over five days in June 2018, where each line represents the PV generation of one house. Green and blue curves correspond to houses facing the east and the west respectively. Although the curves have noticeable differences, there are consistent trends present in the data.
among different houses in the dataset. Additionally, some houses may experience intermittent shadows caused by nearby trees buildings, which can appear and disappear certain times of the day. These shadows are considered as noise, and recorded feature in the dataset provides explicit information about their occurrence or characteristics.
In addition to the meteorological features, we also incorporate autoregressors in our analysis. Autoregressors are advantageous in time-series prediction tasks and aim to capture the temporal dependencies in the data. To determine the autoregressors (PACF). The PACF measures the correlation between lime series and its lagged values while controlling for the correlations with all shorter lags. We select autoregressors with the highest values of the PACF among the lagged values up to two weeks ago. By focusing these highly correlated autoregressors, we aim to capture the most relevant information from the past time steps.
include in our analysis, we utilize the partial autocorrelation function
8.5.2 Polynomial dataset
We examine bimodal dataset where the data for each client generated by sampling a function from one of two GP priors. Each GP prior characterized by a polynomial mean function of order 7 and an SE kernel function. The two modes have distinci polynomial means and length scales associaled with the SE kernel. Additionally, Gaussian noise is introduced into the generated data to account for measurement errors and other sources of variability. Figure illustrates the dataset for total of 24 clients, with 12 clients belonging to the first mode and another 12 clients belonging to the second mode, where each client has 10 training samples.
8.5.3 FEMNIST and EMNIST Datasets
The FEMNIST (Federated Extended MNIST) Caldas et al. 2019) and EMNIST (Extended MNIST) (Cohen et al. 2017) datasels consist of 28 x 28 images of hand-written English characters. Both datasets are used for handwritten character recognition. Specifically, the FEMNIST dataset is employed a 10-way classification task, focusing on the classification of digits. Meanwhile, the EMNIST dataset is utilized for a more comprehensive 62- way classification task. This broader task encompasses the recognition of both lower and upper case English alphabet letters, in addition to digits.
34

---

Published in Transactions on Machine Learning Research (02/2025)
dlents from mode
clients from mode 2
2.0
1.5
0.0
1.0
output
-0.4
0.5
-0.8
0.75
0.50
0.25
0.00
0.25
0.50
0.75
0.75
0.50
0.25
0.00
0.25
0.50
0.75
Input
Figure 6: Polynomial dataset for 24 clients, where 12 clients belong to the first and the rest belong to the second mode. The solid lines represent the unknown true data-generating distribution for each client. Each client has a training dataset with 10 samples. The solid lines in the figure represent the true data-generating distribution, which is unknown to us. The points on the graph represent the noisy samples in each client's training dataset.
FEMNIST identifies the writer for each character, allowing natural partitioning strategy by assigning all images from the same writer to a single client (Caldas et al. 2019). This dataset is heterogeneous due to the inherent diversity in handwriling styles, resulting in skewness within the feature distribution (Tan eet al. 2021). For our analysis, we subsample 40 clients and explore two scenarios: FEMNIST (20), where each client allocated samples and FEMNIST (500), where each client utilizes all available samples, approximately 500 per client.
subsample 10% of the EMNIST dataset that amounts to 81425 total samples. We distribute samples with the same label across 80 clients according to symmetric Dirichlet distribution with parameter 0.4. This approach introduces a skew in the label distribution (Tnn
Following (Wang al, 2020 Marfoq et al. 2021
et al. 2021 thereby synthesizing heterogeneity in the dataset.
8.6
Experiments details
8.6.1
Calibration error
Calibration error (CE) is evaluation metric relevant to probabilistic models. Tt is based the premise that when provided with test input Xj, the model produces probability distribution p(y)(x)) over the predicted target Vj- CE measures the disparity between the predicted confidence intervals and the actual proportions of test data falling within those intervals [Kuleshov et al. 2018 CE is nonnegative measure and a smaller CE value is more desirable.
Calibration error for regression. In defining for regression, we follow the formula proposed by (Rothfuss al. 2021). Denote predictor's cumulative density function (CDF) as f (x,) 1% p(ylx,)dy. Given dataset S {(x, v;)) with samples, compute the corresponding em- pirical frequency for confidence levels S q S ... qu as follows:
1 m
{リ5 (y)) I the == 1.... m
for 1....... H.
If the predictions are well-calibrated, expect that 9n gh proposed by Rothfuss et al. (2021). which formulates CE function of residuals 9n. 9h
We adopt the CE definition
1 H
CE = Σ
H qqn 9h
(41)
35

---

BB Published in Transactions on Machine Learning Research (02/2025)
In our experiments, evaluate to 1.
41 by employing H = 20 equally spaced confidence levels ranging from 0
Calibration error for classification. Given a test input X,, a probabilistic classifier outputs a categorical
probability distribution, P(y; = k x)), where = 1. C, with C' representing the number classes. The classifier's prediction the class label with the highest probability, denoted as ÚJ = arg max f(y = =k] k/xx)) Correspondingly, we define the classifier's confidence in the prediction for the input Xj as Pj = p(y, 9, xx))
For well-calibrated classifier, the confidence is expected to align with the probability of correct classification. For example, with 100 predictions, each having confidence of 0.8, one would anticipate approximately 80 correct classifications (Guo et al. 2017). In an empirical comparison of calibration and accuracy on a dataset S = (;)) following the methodology of Guo et al. (2017), data samples are grouped into = 20 intervals, each of length 1/H, based on their prediction confidence. Let B, == {ilØ, E ("#, #1] for 1,... be the set of indices of points S whose prediction confidence falls within interval h. The accuracy and average confidence of points in Bn are defined as follows:
acc(Br) =
Σ
1 = U₁),
B
jEB
1
cBn)(BB)) =
pj.
B
jEB,
To compare the accuracy and average confidence across all intervals, Guo et al. 2017 calculates CE follows:
CE
Σ
laco( (B.) (Ba)l
(42)
m
This formula measures the deviation between accuracy and prediction confidence level, serving as a metric for assessing the calibration of classification tasks.
8.6.2 Regression experiments details
Baselines details. In the Vanilla approach, the GP hyperparameters for each client are tuned by maxi- mizing the LML of that specific client, without using FL. For pooled GP, we use inducing points to handle computational issucs due to the large number of samples involved in this approach. pFedGP optimizes the average LML across clients oblain the deep kernel hyper-parameters. We adapted pFedGP, originally designed specifically for classification, to suit our regression task. With this adaptation, PFedGP is a special case of our algorithm when using a single prior in SVGD and very wide hyper-prior. We try pFedGP with zero GP mean, originally done in Achituve et al. (2021), and NN mean and do not use inducing points. We apply pFedBayes (Zhang et al. 2022 only to the FEMNIST data the authors only consider classification tasks in their experiments.
parre-prrreeter tuning. We perform 5-fold cross-validation. For all neural networks, we explore structures with the same number of neurons per layer. The number of ncurons per layer can take values of 2n for E 1, 6, and we consider layers. For PAC-PFL, we employ 4 SVGD particles and set k = 4. The parameter 3 is set to the number of samples for each client, B m, To determine the value of r, we search for values greater than 1/(1 + ng)), ensuring that X > + v and thus, satisfying the assumption outlined in Theorem
tuning for all baselines and our method using
4 hidden
The employed hyper-prior is a multivariate Gaussian distribution with a diagonal covariance matrix. It is a distribution over the weights and biases of the mean and kernel neural networks, as well as the noise standard deviation of the likelihood. In all PV experiments, we set the hyper-prior mean for the neural network weights and biases to 0 and the hyper-prior mean for the noise standard deviation to 0.4. These choices help prevent. overfitting.
36

---

Published in Transactions on Machine Learning Research (02/2025)
Table 6: Comparison of probabilistic and non-probabilistcc ( ) FT. approaches along with probabilistic foderrdea basclines on regression tasks. Average RSMSE for all basclines
and CE for
probabilistie approaches
over 5 trials, where + captures 95% confidence interval are reported.
Dutaet
PV-S (150)
PV-EW (150)
PV-S (610)
PV-EW (610)
Polyuomial (10)
Metric
RSMSE
RSMSE
CE
RSMSF
CE
RSMSE
RSMSE
Existing cllents
PAC-PEL.
0.43 + 0.02
0.07 0.00
0.41 + 0.01
0.04 + 0.00
0.42 0.01
0.07 + 0.00
0.40 + 0.01
0.04 4 0.00
0.58 + 0.05
0.14 + 0.02
PFedGP
0.48 + 0.04
0.07 + 0.02
0.45+000
0.06 + 0.01
0.00
0.07 + 0.00
0.46 + 0.01
0.05 0.01
0.80 0.15
0.12+
MTL
0.45 1 0.00
0.43 L. 0.00
0.44
0.41 1 0.00
0.55 1 0.15
MAML
0.57 =
0.03
0.58 0.02
0.52
: 0.13
Vanilla
0.68 = 0.02
0.12+
0.63 0.03
0.12 0.01
0.00
0.04 + 0.00
0.46 = 0.02
0.03
0.73 = 0.07
0.18 + 0.01
Pooled
0.48 +: 0.03
0.26 4: 0.00
0.47 4: 0.05
0.26 4: 0.01
0.49 0.02
0.26 4 0.01
0.46 à: 0.03
0.27 0.01
2.33 4: 0.55
0.33 4: 0.04
PAC-PEL.
0.43 + 0.02
0.07 4 0.00
0.42 + 0.01
0.04 + 0.00
0.03
0.07
+ 0.00
0.43 + 0.01
0.04 4 0.00
0.65 +: 0.02
0.16 + 0.01
New olents
pPddGP
0.46 1 0.04
0.07 4 0.02
0.43 =: 0.02
0.06 1 0.01
0.51 0.05
0.06
0.01
0.46 = 0.02
0.06 0.01
0.87 2: 0.41
0.15
MTL
0.4 0.00
0.12
0.00
0.13
0.88 = 0.12
MAML
0.55 = 0.03
0.52 = 0.01
0.53
0.53 = 0.02
1.11 = 0.28
Vanilla
0.09 = 0.02
0.12 + 0.01
0.03 = 0.03
0.12 0.01
0.69 = 0.02
0.12 # 0.01
0.63 = 0.52
0.12 0.01
0.76 = 0.06
0.24 + 0.03
Pooled
0.48 0.03
0.26 0.00
0.05
0.26
0.48 0.02
0.27 0.00
0.46 0.03
0.27 0.01
2.32 1: 0.37
0.38 0.02
Results. In this section, we evaluate our algorithm and the introduced baselines based on the RSMSE and CE error metrics for existing and new clients. For a given method, we calculate the RSMSE and CE, where applicable, for each existing and new client using a fixed random scod. Then, we compute the average RSMSE and average CE metrics across the existing and new client groups. We repeat this analysis five times using different random seeds and calculate the 95% confidence interval for the sample mean of average RSMSE and average CE.
The results are presented in Table D In each column, the number after the dataset name represents the number of training samples per client. As shown in Table G our method outperforms other approaches in the majority of cases in terms of RSMSE for both existing and new clients. Furthermore, we observe that our method's performance improves with an increase in the number of training samples per client. Concerning CF, PAC-PFT, consistently exhibits low CE values across all scenarios, establishing itself as the top-performing or closely competitive method. This observation aligns with findings in [Rothfuss et al.) (2021), where the authors noted that CE improvement is notable when the meta-learning lasks exhibit greater similarity. In our experiments, given the high client heterogeneity present in both PV-EW and PV-S, the improvements in CE are comparatively modest.
We believe there are three reasons why PAC-PFL. outperforms other baselines. First, by learning the prior with FL and treating posterior inference as personalization, PAC-PFL exhibits a high level of adaptability to individual patterns. Second, PAC-PFL's ability to learn multiple priors enables it to effectively model clients with heterogeneous data distributions. Finally, the regularization of the hyper-posterior towards the hyper-prior in PAC-PFT. helps prevent overfitting and allows for learning more complex prior means even with limited training data. For example, when selecting the GP prior mean structure through pross-validation it is observed that the best GP mean for PFedGP is linear function, while for PAC-PFL it is a 2-layer neural network with 32 neurons per layer. The use of a more complex prior mean in PAC-PFT, allows for the potential capture of more intricate patterns in the data.
8.6.3 Classification experiments details
Baselines implementations. We implemented pFedGP Achituve al. (2021 | using the code provided by the authors without utilizing inducing points. Since the source code for paydBayes has not been publicly released, we rely on an unofficial implementation 0 Our implementations of FedAvg, MTL, and MAML are
based on the code provided in Xie et al. 2023
tuning. Hlyper-parameter tuning for all basclines and our method is conducted using the cross-validation approach. In the case of PAC-PFL, we select ,3 and 7 following the same procedure as employed for the PV dataset. We use 3 SVGD particles and a multivariate Gaussian hyper-prior with a diagonal covariance matrix.
9https//ghthub..cmm/ BAll/eeBeB/e/////aaaaa
37


Reviews:
[{"review_id": "1KZO8GHfrK", "reviewer": "Reviewer_J4a8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper does a good job describing the key challenges in the PFL domain. \n2. The key idea to adapt meta-learning to PFL is interesting.\n\nWeaknesses:\n\n1. From a theoretical point of view, I think there is a lack of real novelty. Instead the paper heavily draws from prior art and the novelty is incremental at best. I am not an expert in the area, but it would be greatly helpful if you could clearly articulate the key challenges involved in adapting meta-learning ideas to the PFL domain. Separate out the new techniques from old techniques.\n2. While the premise is quite interesting, the claims in the introduction are not substantiated to my understanding. For example, how are you demonstrating that extreme heterogeneity is tackled better than prior art? Further, the datasets are quite tiny, how do the techniques scale with larger datasets. Even so, there are only 3 datasets on which the approach is evaluated and only on one dataset the approach seems to beat meta learning approaches (which is not surprising the approach is borrowed from it). \n3. Scalability is explicitly called out when discussing the work by Corinzia & Buhmann (2019). How is PAC-PFL better than that? It should be noted that while the primary concern seems to be \"sequential\" nature of fine-tuning models at the client, this can effectively be done in parallel, so I am not sure what the authors mean by \"scalability\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work considers the problem of PFL (personalized federated learning) in a heterogenous settings. The proposed approach aims to address the following issues in PFL: a) lack of uncertainty quantification (which is address by using probabilistic models) b) inability to capture heterogeneity (addressed using a notion of \"hyper-priors\") c) performance degradation when client datasets are small  d) enabling inclusion of new datasets. The central idea is to adapt ideas from PAC-Bayesian meta-learning to PFL settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Firstly, a clear demarkation needs to be done between existing work and the new approach being proposed. What is the net new of the work, this is not clear. \n2. The experimental needs to demonstrate the issues discussed in the introduction -- particularly the heterogeneity, small client performance degradation etc. It needs to be a lot more extensive, in-lieu of theoretical results. \n3. Define hyper-prior and hyper-posterior formally (perhaps within a definition environment)\n4. Address the scalability blurb in the related work -- It is written in a hand-wavy fashion in my opinion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0WBr7jW2je", "reviewer": "Reviewer_bAQB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The experimental section seems strong, though I am not aware of the latest literature. (Given the claims please make sure the section is as thorough as possible.)\n* Table 1 is a nice summary\n\n**Weaknesses**\n* Notation is hard to follow. I had difficulty reading this paper. Maybe more figures would help?\n* Presentation is poor.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies personalized federated learning. The PAC-PFL method is introduced, where a shared (hyper)-posterior is learned and each client determines their posterior. The benefit of this approach is that there is no regularization towards a shared model, which enhance personalization. Furthermore, it can easily adapt to include new information. The authors claim that PAC-PFL is sota across all the datasets they evaluate.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would it be possible to re-work this paper to improve presentation? There is a lot of jargon. Even with an understanding of what all these words mean, it isn't very clear. prior of what? posterior of what? What makes something a \"hyper\"-posterior?\n\nThe notation is very cumbersome, is there anyway this can be improved?\n\nAdd some suitable parenthesis to Eq. 3\n\nDo the results depend on the degree of heterogeneity?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nyb6HfydDc", "reviewer": "Reviewer_xC1T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The propose approach can handle highly heterogeneous clients by formulating a hyper-posterior, enhancing the flexibility to capture diverse patterns compared to methods with a single shared prior. It adapts well to individual client patterns by treating posterior inference as personalization after learning the prior with FL.\n2. Theoretical results are established, providing guarantees on the performance and generalization ability of the model.\n3. Experimental results show the proposed approach outperforms various baselines.\n\nWeaknesses:\n1. It would be better to provide more motivation / real applications of PFL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents PAC - PFL, a novel algorithm addressing challenges in federated learning. It learns a shared hyper-posterior federatively for personalized inference, mitigates overfitting, and provides generalization bounds for new clients. Experiments show its effectiveness in accurate and well-calibrated predictions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Provide more motivation / real applications of PFL.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work addresses the challenge of Personalized Federated Learning (PFL) in heterogeneous settings. It proposes adapting ideas from PAC-Bayesian meta-learning to develop a novel algorithm for PFL. The proposed approach is specifically designed to handle high client heterogeneity and mitigate overfitting to a central model, mainly when client datasets are small. Additionally, the method is robust to dynamic scenarios, allowing clients to join the training process at later phases rather than all simultaneously.

The reviewers commended the paper for its innovative approach, which bridges two distinct branches of machine learning research. However, they also noted that further investigation is needed to verify whether the claims regarding scalability and extreme levels of heterogeneity hold beyond the ranges studied in this work.
Decision: Accept (poster)