FedPop: A Bayesian Approach for Personalised Federated Learning
Authors: Nikita Yurevich Kotelevskii
Maxime Vono
Alain Durmus
Eric Moulines
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bayesian inference
latent variable
MCMC
federated learning

Abstract: Personalised federated learning (FL) aims at collaboratively learning a machine learning model tailored for each client. Albeit promising advances have been made in this direction, most of the existing approaches do not allow for uncertainty quantification which is crucial in many applications. In addition, personalisation in the cross-silo and cross-device setting still involves important issues, especially for new clients or those having a small number of observations. This paper aims at filling these gaps. To this end, we propose a novel methodology coined FedPop by recasting personalised FL into the population modeling paradigm where clients’ models involve fixed common population parameters and random effects, aiming at explaining data heterogeneity. To derive convergence guarantees for our scheme, we introduce a new class of federated stochastic optimisation algorithms that relies on Markov chain Monte Carlo methods. Compared to existing personalised FL methods, the proposed methodology has important benefits: it is robust to client drift, practical for inference on new clients, and above all, enables uncertainty quantification under mild computational and memory overheads. We provide nonasymptotic convergence guarantees for the proposed algorithms and illustrate their performances on various personalised federated learning tasks.

central server FedPop: A Bayesian Approach for Personalised Federated Learning
Nikita Kotelevskii
Maxime Vono" Criteo Al Lab Paris, France www.oorritto..... com Equal contribution
Skolkovo Institute of Science and Technology
Moscow, Russia
Nikita. Kotelevski @skol tecollleccc... ru Equal contribution
Alain Durmus
Eric Moulines
ENS Paris-
Ecole Polytechnique
alain, cerrmu@@ens-parlee paris-saclay.............aaa.a..
eric. www.mmuulneessem hneeooonliisepllyyyycccniiuuuuu edu
Abstract
Personalised federated learning (FL) aims collaboratively learning a machine learning model tailored for each client. Albeit promising advances have been made in this direction, most of existing approaches do allow for uncertainty quantification which is crucial in many applications. In addition, personalisation in the cross-silo and cross-device setting still involves important issues, especially for new clients those having small number of observations. This paper aims filling these gaps. To this end, we propose a novel methodology coined FedPop by recasting personalised FL into the population modeling paradigm where clients' models involve fixed common population parameters and random effects, aiming at explaining data heterogeneity. To derive convergence guarantees for our scheme, we introduce a new class of federated stochastic optimisation algorithms which relies Markov chain Monte Carlo methods. Compared to existing personalised FL methods, the proposed methodology has important benefits: is robust to client drift, practical for inference quantification under mild computational and memory overheads. We provide non- asymptotic convergence guarantees for the proposed algorithms and illustrate their performances on various personalised federated learning tasks.
clients, and above all, enables uncertainty
Introduction
Federated learning (FL) is a recent machine learning paradigm in which distributed clients holding siloed data collaborate in solving a learning problem, usually under the coordination (Wang et al., 2021; Kairouz et al., 2021). One of the main focus of FL is on so-called cross-device applications where large number of personal electronic devices such as mobile phones, wearable devices or home assistants collect and store data at the edges of decentralised network (McMahan al., 2017).
While standard FL methods (McMahan et al., 2017: Alistarh et al., 2017; Karimireddy et al., 2020; Horváth et al., 2019; Li et al., 2020) have focused on training global model that be applied to individual agents, the relevance of such inferences has recently been questioned due to statistical heterogeneity between clients. Indeed, the considered common model may generalise well on a client with a local data distribution that differs significantly from the global data distribution, especially if that client has not participated in the training process. In fact, might even be better for such clients to derive a local model from their own data set. To circumvent these issues, number of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

personalised federated learning approaches have recently been proposed, that use local models to fit client-specific data distribution while capturing some shared knowledge via a federated scheme (Tan et al., 2022). Personalisation has previously been approached using multi-task learning (Smith et al., 2017), meta-learning (Jiang et al., 2019; Khodak et al., 2019), client clustering (Briggs et al., 2020), data interpolation (Mansour et al., 2020), model interpolation (Hanzely and Richtárik, 2020; Hanzely et al., 2020) or partially local models (Singhal et al., 2021: Collins et al., 2021). While these methodologies are promising, they only partially solve the personalisation problem in highly heterogeneous federated settings and have no means of quantifying uncertainty. In addition, cross- device FL also faces other important challenges such (extreme) partial device participation, small local data sets, limited upload bandwidth and device capabilities (Kairouz et al., 202 1). Addressing these problems for personalised FL requires new paradigms regarding how model knowledge is shared and personalisation is performed locally.
Proposed Approach. In this paper, we adopt novel perspective to model the problem of personalised FL. This paradigm, called rixee-effects modeling (also known as multi-level or population approach) is widely used to analyse data that have clustered or nested structure, as in medical or biological research where multiple observations per patient are available (Gelman and Hill, 2007: Long, 2011; Lavielle, 2014). Although the hierarchical structure of FL has already been noted (Plassier et al., 2021; Grant et al., 2018; Hong et al., 2022), the mixed-effects paradigm has interestingly never been considered. Leveraging this framework, we develop FedPop and propose an efficient computational solution to perform inference under this model. More precisely, we introduce a novel class of federated stochastic approximation algorithms based on parallel Markoy Chain Monte Carlo (MCMC) methods. In the proposed framework, we also pay special attention to the cross-device setting by taking into account partial client participation, and by addressing the communication bottleneck with both multiple local updates and the use of lossy compression operators.
model for personalised FL called
Benefits. Up to the authors' knowledge, FedPop is the first personalised FL approach that allows cheap uncertainty quantification with a poooretically-gronndee
methodology. The proposed frame- work also comes with other interesting properties. First, in contrast to most of personalised FL methods that only consider "fixed-effects" models (Collins et al., 2021; Hanzely et al., 2021; Smith et al., 2017). FedPop provides credibility information (via credibility regions) and allows more accurate inference for clients with small and heterogeneous local data via partial pooling (Gelman and Hill, 2007). In addition, inference for new clients which did not participate in the training phase can be easily performed by sampling from the prior over the local random effects. Second, contrary to existing Bayesian FL approaches that aim to provide credibility information by sampling from a target posterior distribution (Hong al., 2022: Yoon et al., 2018: 8: Vono et al., 2022; El Mekkaoui al., 2021). FedPop allows to perform personalisation and cheaper on-device uncertainty quantification taking an empirical Bayes prediction approach. Finally, an important benefit of FedPop its ability to allow for multiple local updates without suffering from the client- drift phenomenon (Karimireddy et al., 2020).
Outline and Contributions. Our contributions are fourfold. First, in Section 2, we propose a novel probabilistic methodology, which we call FedPop, address personalisation under the cross- device FL paradigm. To perform efficient inference under this model, we introduce a general class of stochastic approximation algorithms based on MCMC. Second, we provide in Section 3 non- asymptotic convergence guarantees for the proposed methodology. Then, we perform in Section 4 a comparison between the proposed approach and exisiting works. Finally, we illustrate in Section the benefits of our methodology on several federated learning benchmarks involving both synthetic and real data.
2 Proposed Approach
In this section, we present the statistical estimation problem we are considering and the proposed methodology called FedPop to address it.
Problem Formulation. We are interested the cross-device FL setting involving a large number b E N of clients, potentially unreliable i.e. not necessarily available at each communication round. These clients are assumed to own sensitive local data sets (e))e( In this framework, we aim to make both uncertainty guantification and personalised statistical inference by learning a local model

---

tailored to each client. To this end, and inspired by the population approach used in the biological and physical sciences (Lavielle, 2014), we consider mixed-effects modeling for each client leading to the local marginal likelihood function defined, for any E [b], by
P(D 0, - - p(D; Φ 20) ()dz())
(I)
where + Re stands for fixed effect and {2() HE Z, I R represent random effects aimed at explaining statistical heterogeneity between local data sets {Di)eeo) The objective of the fixed (i.e. constant across all clients) part to capture common representation (e.g. same features across different classes of images) while the random part, which is typically oww-dimessional, performs personalisation and is assumed be drawn from a population prior whose variance aims at modeling data heterogeneity.
D2
B
Figure illustrates this statistical framework, referred to as FedPop, by showing its directed acyclic graph (DAG) where grey-filled shapes indicate observed variables, white-filled shapes unknown variables and squared shapes variables to be estimated.
Figure 1: DAG for FedPop.
When the size of the local data set D; is small, this common prior leverages information from other clients to limit the risk of overfitting and is often called partial pooling in the multi-level statistical literature (Gelman and Hill, 2007, Section 12). Examples of model architectures involving e and (2()) Je( include for instance composition- -based architectures p(D; Φ. 2()) = p(D; lh hgg)) where bs and hg are two neural networks (Collins et al., 2021; Arivazhagan et al., 2019). For the sake of generality, we propose to adopt flexible energy-based prior distribution of the form for each i [6].
p(2(0 133) ((3) exp ( EE(( where Z(3) = / exp { EE(= dz
Here, Z(3) is a normalising constant and E-: B) represents an energy function, typically neural network, parameterised by set of parameters B E C Rdd (LeCun et al., 2006). This framework is particularly interesting in the cross-device setting where the number of clients is large it allows for efficient enrichment of the model. However, in the case where is small, the inference of the parameter is difficult. In this situation, a more pragmatic solution is consider common prior for the local random effects (2(() Fee(o) which is held fixed, i.e. ( () () expf-E(E exp(-E((2))} for any 3 E B. Finally, for completeness, we allow the use of prior model p db 3) for the hyperparameters 9). Using Bayes' rule (Robert, 2001) and by denoting D L D; the global data set, the posterior distribution associated with these hyperparameters admits probability density function which can be written as
/ P(D, I o, (0, 2())(( 113)z
( DD x p(o, 8) II
Set 0= = ( (o. 8} E θ with = B. In the sequel, we will be interested solving the maximum a posteriori problem given by
l" E arg max logp(e, (D),
(2)
BEB
Σ log / p(D, &, (z(()))))) 18)dz(0 + +C,
log 1gg,( D) (0, 3)
where C' R is constant independent of Once have estimated 6* using an empirical Bayesian approach, we can perform "for free" on-device uncertainty quantification for each client i E [b] by sampling from the local posterior distribution p(2(1) D. o", f*), which is typically designed to be low-dimensional.
Algorithm. solve the optimisation problem (2), we can either use an alternating maximisation algorithm or perform global maximisation over θ, Since the former approach requires more upload
3

---

bandwidth, in this work we consider the second alternative which is more suitable for FL. The gradient of the objective function (3) being intractable, we propose resort to the stochastic approximation framework (Robbins and Monro, 1951) which iteratively defines (1h, 3k k,E,,, starting from any (do. (B0) E O via the recursions for any ls E N,
3K+1 Il
3x +
(1) ME+1
V log p 3c)
+
Σ
8k
(ok 3xc)
Øk+1 = 11g
Φx + 1
(2)
V log log 3k)
+
hk (Ōx: 3k)
where Ilc denotes the projection onto Ce {2, B). (m 7/2 KEN are sequences of step-sizes, and (6) i E [b] k ] and (h e [b] k E N*) estimators of the intractable gradients .) and (),3) + Ve log o, 3) at (øx, 8k), where p(D; Φ, B) is defined in (1) for any [b].
(e, 3) + Va log
The choices of the estimators {g. : i b N*) and (ng : [b, E N* are motivated by the Fisher identity. More precisely, under mild regularity assumptions, and using the Lebesgue dominated convergence theorem, we have for any, (6,3) E 0, e [b]
V log pD; 0) = / V 20
Φ, B) p(z(0) Di, D, B)dz
V log PD; (0 = { V PD (0) ( p(2(0)
D., 3),z(9
which suggests to consider
M
( 8k (), M m=1
V 9 log p( p(z (8.m) 18),
(4)
( (0. B)
V log p(D;
z (( 0),
(5)
M
where M E N and z (1.1)M)) (Z(h m are approximate samples from p(2(1) D;, o, 8). 3.0 7 E (0.5),0 E θ} where for any step-size 7, 0,0% Markov kernel which targets close approximation of p(2(6) D., 0) with 0 = (c) 3). As an example, we can use overdamped Langevin dynamics (Roberts and Tweedie, 1996; Welling and Teh, 2011) to generate these samples. In this case, Q is associated with Gaussian probability density
More precisely, we consider a family (0
function 27.8 (2()), ,-) with mean ( V log p(2(0) Di, 0) and variance 2-1a- Note that the number of Monte Carlo draws per iteration is considered constant here but can easily generalise our scheme to the non-constant setting. In addition, our scheme can also be generalised by taking into account stochastic gradient estimators of (4) and (5). For the sake of simplicity, we present our approach with standard gradients.
(8)
In this framework, we present the main steps of the corresponding stochastic approximation algorithm, called FedSOUK, in Algorithm 1. Since we consider the cross-device federated setting, note that only random subset Ak+1 of active (i.e. available) clients communicates with the central server each iteration l E N. In addition, due to limited upload bandwidth, the potentially high-dimensional gradient estimator (5) is compressed locally via unbiased stochastic compression operator 6x+1 before being sent to the central server (Alistarh et al., 2017; Philippenko and Dieuleveut, 2020).
Stateful and Stateless Versions. Depending on local memory constraints and the participation rate, we allow for a possible warm-start strategy across communication rounds to improve the convergence propertics of the proposed algorithm so that the proposed algorithm becomes stateful, see steps 4 - 7 in Algorithm 1. In cases the participation rate is very low (e.g. each client might only participates once to the training process), we replace this warm-start strategy by the initialisation z(,.)) p(- B&) for any E [b] and l E (O...., K 1}. This yields a stateless version of our algorithm more suitable to the cross-device setting. Obviously, compared to the previously proposed warm-start
4

---

strategy, the performances of Algorithm will be affected negatively if we are using the same number of local iterations M. We end up with an interesting trade-off between local computations and communication: if client-server communication is bottleneck, the stateless version of the algorithm allows to reduce the communication overhead at the price of longer sampling procedures on each client. Such analyses will be illustrated empirically in Section 5.
Computation Complexity. Compared to standard FL methods, our approach has an additional
computational cost on the client side associated with Monte Carlo approximations (7&) and (J&) ). In practice, this cost negligible. Indeed, in our experiments, results in terms of accuracy on the test dataset. M E [1. 10) was sufficient to obtain state-of-the- We would like to emphasize that this additional computational cost has also two side advantages compared to existing FL approaches: (I) it allows us to communicate less frequently with the central server and (2) it allows converge faster when the number of local iterations increases since Monte Carlo approximation becomes better.
found that using small value of
Communication Overhead. As pointed out in Table our methodology FedPop improves upon existing FL approaches regarding the communication overhead. Indeed, FedPop offers the flexibility use both compression for sending updates to the server, and multiple local steps reduce the communication frequency. As such, depending on the bandwidth and local computational power, the practitioner adapt the number of local iterations and the parameter of the compression operator. Up to our knowledge, this work is the first one combining compression and multiple local steps for personalised FL.
Robustness to client drift. For simplicity, we will take the example of FedSOUL (see Algorithm I) which uses the Markov kernel associated with Langevin Monte Carlo to compute gradient estimates of the local marginal likelihood. However, our answer holds for general Markov kernels (adjusted or unadjusted). In this scenario, M steps of Langevin Monte Carlo are performed
each device draw samples {Z (tm)) used to compute Monte Carlo estimates ( and (J{() J. Increasing the number of local steps M does not slow down convergence but instead allows for more accurate Monte Carlo
integration and hence better convergence properties. contrast, the client drift phenomenon for
classical FL approaches (e.g. FedAvg proposed in McMahan et al. (2017)) slows down convergence as the number of local iterations M increases.
Simple inference on new clients. Typical personalized FL. approaches such as DITTO or FedRep require additional local training for inference on new clients. contrast, the proposed methodology FedPop allows for a cheaper two-step approach once we have estimated 0* = ,²), as detailed below for new client + 1 with feature vector æ:
1. Sample (2 he(() in i.i.d manner.
2. Estimate the posterior predictive function by p(- (x) L El p(. 10t 2 2011: x).
The prior p(=++1 (18*) is typically chosen so that sampling is computationally cheap, egg. Gaussian with diagonal covariance matrix as in our experiments, see Section 5.
3 Theoretical Guarantees
In this section, we present non-asymptotic convergence guarantees for Algorithm I when the family of Markov kernels (Q 7 E 70, E O, [b} is associated to unadjusted, i.e. without Metropolis acceptance step, overdamped Langevin dynamics (Durmus and Moulines, 2017; Dalalyan, 2017). The bounds we derive allow to showcase explicitly the impact of FL constraints, namely partial participation and compression. Results for general unadjusted Markov kernels are postponed to the supplement.
To show our theoretical results and resort to standard assumptions made in the stochastic approxima- tion literature, we consider a minimisation problem and rewrite the opposite of the objective function
(3) for any 0 θ as
b
((0) - b- Σ ((0),
where fi (0) = - log 9) b blogp (D; 10,10,80 ()
(6)
1=1
5

---

Algorithm FL via Stochastic Optimisation using Unadjusted Kernel (FedSOUK)
1: Input: nb. outer iterations K, nb. local iterations M, Markov kernels (0 7,00), 7,0 step-sizes
{Mh Th se and initial points zØ" E R., Bo B and do Φ.
(1), (2)
2: for k = 0 to K - do
3: Server sends (5K+1, to clients belonging to Ak+1-
4: for i E Ax+1 // On active clients Ak+1 do
5:
11 W/Wammsstar
of the SA scheme if possible
6:
if k: 2 1 then
Set (2.0) (i)
7:
8:
end
9:
// Computation of key quantities using MCMC for m = 0 to M do
10:
Draw z z(im++1) Q
(2)
11:
7.0 k
12:
// For Langevin dynamics
// Draw Ss elim+1) ~ N(Oa, Ia).
13:
// Set 2 z((mn+1)_ = 2 (l.m) + yV, log p(z ((im))
Di, Bk) + 227c (im+
14:
15:
end for // Communication cation with the server
16:
17:
Set [ Mi 2m=1
V logp
2 (()m)))
BE 0x)
18:
Set 1 = 1 EM V log
D; (Lm) &
NT
19:
Send r/ and Ck+1
(J(x)
to the central server.
20:
end for
21:
Set Bk+1 1Tg [B + 11++1
(1)
V log plok, + |Anial EiEAm, T(7]). V a log (b,, 10b,, 3x) + Metil EiErhh Ckt1 (JK
22:
Set 0k+1 = Io
(x + 1k+11
end for Output: fØx, {x} and samples {Z ((ib)mm) M
24:
K-1 /m=1.
Non-Asymptotic Convergence Bounds. For the sake of better readability, we only detail in the main paper assumptions regarding the objective function, compression operators and the partial participation scenario. Technical assumptions related to the Markov kernels (Q ) are postponed to ergodic condition and can provide samples sufficiently close to the local posterior distribution p(z(1) D;, 0). For the sake of simplicity, also assume that for any ls N ,1( 11 Mi, see Algorithm 1.
the supplement. In spirit, we require, for any i [b], 0 θ and % that Q satisfies
(2)
We make the following assumptions on @ and the family of functions {ff : E [b]].
HI. θ is convex, closed subset of de and θ B(0, Re) for Re > 0.
H2. For any i E [b], the following conditions hold.
(i) The function fs defined in (6) is convex.
(ii) There exist an open set U E Id and Lj > such that C U. CU, and for any
01.0 Og E A
[V2) V f. - L, 10 10:-110000 - 01
The assumption below requires compression operators {36x kEEN- to be unbiased and to have bounded variance. Such an assumption is for instance verified by stochastic quantisation operators, see Alistarh et al. (2017).
H3. The compression operators {EK)NEN- are independent and satisfy the following conditions.
(i) For any l E N TRd, (((E(1)) =
(ii) There exists 2 1, such that for any l E N E Rd. (l)( =11 s w 1|ull

---

We finally assume that each client has probability p E (0, 1] to be active at each communication round. We would like to point out that this partial participation assumption can be associated to specific compression operator satisfying H3.
H4. For any l E N", A = {i E [b] : Bi = 1] where for any i E [b]. [Br, k E is a family of i.i.d. Bernouilli random variables with success probability E (0, 1|.
Under these assumptions, the next result establishes that (0x)keN defined by 0g Σj=1 1/0//EEi=1 n7)) converges towards an element of arg ming f.
Theorem 1. Assume HI-H4 along with A8 detailed in the supplement and let for any k E [K] Th E ( Then, for any K E EN,, we have
E Th m&(((()) - -((-)}
EK
E [f(Ot) f(0.)] -((0)) S
S A(r)
Σ 7K
Lk TK
where EK depends linearty on (w/p) Ek Thi and A(7) = Ch with a > O and C' is independent of w. p and (mx)) Closed-form formulas for these constants are provided in the supplement.
An interesting feature of Algorithm is that convergence towards a minimum of f is possible and
the impact of partial participation and compression vanishes when limgg=oo A2 0. More precisely, lim = EK/ (E my) and lim A(7) = 0 which shows that we can tend towards a minimum of f with arbitrary precision 0 by setting the step-size yy to a small enough value.
4 Related Works
As pointed out in Section 1, many different approaches have been proposed to address personalisation and uncertainty quantification under the federated learning paradigm. This section reviews the main related existing lines of research and shows that the proposed methodology provides many benefits; see Table 1. Interestingly, we also show that FedPop encompasses some of the existing FL models,
Bayesian FL. One of our main motivations is the possibility to perform grounded uncertainty quantification in FL by resorting to the Bayesian paradigm. In the recent years, many works have suggested to adapt serial workhorses stochastic simulation approaches such as MCMC or variational inference to the FL setting (Chen and Chao, 2020; Liu and Simeone, 2021b,a; Vono et al., 2022: El Mckkaoui et al., 2021; Corinzia et al., 2019; Bui et al.. 2018; Plassier et al., 2021; Deng et al., 2021a). Although some of these approaches address important FI. challenges such as the communication bottleneck, partial participation or limited computational device resources, they are not suitable for uncertainty quantification in the cross-device FL. scenario. Indeed, all these approaches assume that the posterior distribution targeted by each client is parametrised by a single potentially high-dimensional parameter of size d4 + d, see (I). This prevents a sufficient number of samples from being stored locally to perform uncertainty quantification and Bayesian model averaging, especially when the model is a large neural network. In contrast, our approach decouples this unique high-dimensional parameter into fixed part e and a low-dimensional random part 21) significantly reducing the memory footprint of local sample storage.
In addition, Bayesian FL methods aim at sampling a random parameter from a target probability distribution where (0) x - where f = (1/b) Σ St denotes the negative log- likelioel associated to the i-th client. On the other hand the proposed methodology considers a mixed-effects modeling approach where parameters are divided into two categories: fixed component and random one for each client, As such, the mixed-effects approach is in essence an empirical Guyesian/marginne likelihood approach (Casella, 1985; Robbins, 1992). It corresponds to a hierarchical model that aims to combine the modeling flcxibility and uncertainty assessment of Bayesian inference with computational pragmatism. More precisely, part of the parameters (fixed-effects are estimated via marginal likelihood maximisation and the rest (random effects) using common Bayesian techniques, which are in most cases low dimensional. As result, up to our knowledge, the model and approach that we propose is novel in FL and comes with many benefits as shown in Table 1.
Personalised FL. Beside uncertainty quantification, we also aim at providing each client with dedicated personalised model. Among the numerous existing personalised Fl. approaches, those related to FedPop can be broadly classified into two groups: meta-learning and partially local
7

---

Table 1: Overview of the main existing personalised FL (top rows) and Bayesian FL (bottom rows) approaches related the proposed framework. Column "Pp" refers to partial participation, "perso." to personalised approaches, "bounds" to available convergence guarantees, "UQ" to available uncertainty quantification, "com." " to the scheme (multiple local steps and/or compression) used to address the communication bottleneck and "memory" to the client memory footprint where M stands for the number of samples.
METHOD
PP
PERSO.
BOUNDS
UQ
COM.
MEMORY
FEDPOP INSTANCE
PER-FEDAVG
LOCAL STEPS
+ de
PFEDME
X
X
LOCAL STEPS
d + dp
X
FEDREP
/
V
X
d + dx
DITTO
V
X
LOCAL STIPS
d do
LG-FEDAVG
V
LOCAL STEPS
d de
QLSD
V
X
COMPRESSION
M(d de)
FSGLD
X
X
LOCAL STIPS
M(d + de)
FEDBE
V
X
LOCAL STEPS
M(d + de)
X
DG-LMC
X
X
M(d + de)
FEDPOP
V
BOTH
Md + do
methods. Meta-learning based FL. methods aim at training global model conducive to fast training of personalised models. Such goal can be achieved, for example, by local fine-tuning (Fallah et al., 2020), regularisation of local models towards their average (Hanzely and Richtárik, 2020; Hanzely al., 2021) the opposite (Li et al., 2021), and model interpolation (Liang al., 2019). On the other hand, FL methods based on partial decoupling take an approach similar to ours by splitting the initial model into backbone component and local one aimed personalisation (Collins et al., 2021; Arivazhagan et al., 2019; Pillutla et al., 2022). This partial decoupling could also enhance privacy as discussed in Singhal et (2021). The main difference with FedPop is that such approaches based on empirical risk minimisation cannot provide credibility information.
FedPop: A Compromise between Standard and Personalised FI.. Interestingly, show here that the FedPop framework allows existing FL approaches to be retrieved in certain regimes. To this end, we assume that the prior ((2(() 3) is Gaussian with mean j and covariance matrix I²I, so that [3 {u, J.. If - 0+, then this Gaussian prior tends towards the Dirac distribution centered at 1a. and the local likelihood becomes p(D; d, (4), which corresponds to the local objective of standard FI. approaches such as FedAvg (McMahan al., 2017). On the other hand, when -> oc, no common information 3 is used to locally regress 2(0 and we end up with the FedRep algorithm (Collins et al., 2021). This shows that FedPop stands for a subtle compromise between standard and personalised FL which should benefit clients with small data sets by pooling information via common prior. Finally, in the extreme scenario where e is the null vector, our approach amounts to the Bayesian FL approach DG-LMC proposed in Plassier et al. (2021).
Numerical Experiments
In this section, we illustrate the benefits of our methodology on several FL benchmarks associated to both synthetic and real data. Since existing Bayesian FL approaches are not suited for personalisation (see Table 1), only compare the performances of Algorithm with personalised FL methods. In all our experiments, we use overdamped Langevin dynamics to sample locally and call this specific instance of Algorithm 1, FedSOUL. In addition, we set p(=(0) (3) N(u, o²1a) with 3 = [14, a for simplicity. To be comparable with existing personalised FL approaches that only consider periodic communication via multiple local steps, we do not resort to the proposed compression mechanism although the latter could be of interest for real-world applications. Additional experiments and details about experimental design are provided in the supplement.
Synthetic Data. We start by showcasing the benefits of FedSOUL for clients having small and highly heterogeneous data sets as pointed out in Section and Section 2. To this end, we consider a similar
experimental setting as in Collins et al. (2021) where synthetic observations (v) ]E(N D,

---

generated via the following procedure: o7))) N(Ok: T&) and v N(=ine o True 0.1..
The ground-truth parameters (6)
ztrue E Rd and Otrue lxxd have been randomly generated beforchand with kd, = (2,20). Compared to Collins et al. (2021), we use heterogencous data partitions across clients that 90% of the b 100 clients have small data sets of size 5 and the remaining 10% have data sets of size 10. We compare our results with FedRep (Collins al., 2021) and FedAvg (MeMahan al., 2017) since they stand for two limiting in- stances of the proposed methodology. see Section and Gelman and Hill (2007, Section 12).
Figure 2 compares the different ap- proaches by computing the princi- ple angle distance' (respectively the
12 nerm the estimation crror
Princical Angle Dizance
30
l2 norm) between Perue (respectively (() and its estimated value; the lesser the better. In contrast to its main competitors and based on both met- rics, FedSOUL provides an impressive improvement. This illustrates the ben- efits of the introduction of a common prior p( ((2()) (3) which allows pre- vent from overfitting on clients with small data sets while performing personalisation. Additional results with other choices for (b, d, k) and data partitioning strategies are available in the supplement.
PesTap
Tedtag
PotSOUL
Redkep
Redlwg
250
300
500
Figure 2: Small data sets synthetic data.
Moreover, to compare our algorithm with a non-FL setting, we perform a non-distributed and non- federated stochastic approximation algorithm to find 6" using large number of iterations get an accurate approximation of the optimal parameter 0*. Then, we use FedPop to obtain estimate 0" and measure the relative error in 22- distance between l" and Ø* For some outer iterations T 100, the relative error was less than 10-3 which illustrates the relevance of our theoretical results. We also the performances of the proposed approach when the warm-start strategy is not used. In this case, we have to set M = 50 to achieve the same performances as in the stateful variant of FedSOUL.
real image data sets, namely CIFAR-10 10 and CIFAR- 100 (Krizhevsky, 2009). For our likelihood model defined by p(D; Ø, ()), we use 5-layer convolutional neural networks and perform personalisation for the last layer. We set 6 = 100 for convenience and control data heterogeneity by assigning to each client N; images belonging to only S different classes.
Real Data. We consider
Small data sets. Under this setting, we first consider (10% 50%, 90%) of clients having small data sets of size either N, = or N₂ = 10; while remaining clients have larger data sets of size N; = 25. We com- pare our approach with FedRep since it stands for the state-of-the-art person- alised FL approach. The algorithms are trained fulfilling the same com- putational budget. Figure 3 shows the average accuracy across clients for the two approaches on both CIFAR- 10 and CIFAR- 100. We can that FedSOUL is consistently better than FedRep over different configurations.
CIN
C3
10
12
Figure (right) CIFAR-10 with S and (left) CIFAR- 100 with S = 20. The r-axis refers to the percentage of clients having N; E (5, 10) images.
Full data sets. In addition to show that the proposed approach achieves state-of-the-ar performances on small data sets (which is common in the cross-device scenario), we now illustrate that FedSOUL is also competitive on larger data sets. To this end, we use all training images in CIFAR-10 and CIFAR- 100 image data sets and consider the same data partitioning as in Collins et al. (2021). More precisely, in this case the number of observations and the number of classes per client are uniformly shared over the clients. Table shows our results in comparison with ofate-of-the--e art personalised that that model outperforms other methods on both CIFAR-10
FL approaches. We
defined in (Collins et al., 2021, Detinition 1)

---

Table 2: Real data Full data sets. Accuracy (in %) on test samples, FedAvg and SCAFFOLD are not personalised FL. approaches but stand for well-known FL benchmarks.
CIFAR-I0
CIFAR-100
clients b, classes per client S)
(100, 2)
(100, 5)
(100, 5)
(100, 20)
Local learning only
89.79
70.68
75.29
41.29
FedAvg (McMahan et al., 2017)
42.65
51.78
23.94
31.97
SCAFFOLD (Karimireddy et al., 2020)
37.72
47.33
20.32
22.52
LG-FedAvg (Liang et al., 2019)
84.14
63.02
72.44
38.76
Per-FedAvg (Fallah et al., 2020)
82.27
67.20
72.05
52.49
L2GD (Hanzely and Richtárik, 2020)
81.04
59.98
72.13
42.84
APFL (Deng et al., 2021b)
83.77
72.29
78.20
55.44
DITTO (Li al., 2021)
85.39
70.34
78.91
56.34
FedRep (Collins et al., 2021)
87.70
75.68
79.15
56.10
FedAvg fine-tuning (FT)
85.63
71.32
79.03
56.19
FedSOUL (this paper)
91.12
79.48
79.56
59.73
and CIFAR-100 by large margin. Additional results with other personalised FL algorithms postponed to the supplement.
Uncertainty Quantification on Real Data. As highlighted in Table 1, one advantage of the proposed approach compared to existing personalised FL methods is the ability to perform uncertainty quantification by sampling locally from the posterior ((2 D,, ØK 5x), see Algorithm We illustrate this feature by computing on CIFAR-10 calibration curves and scores (e.g. expected calibration error aka ECE) on specific client; and by performing an -iit--ddistibutiion analysis on UNIST//FahionMNNI data sets. Figure 4 shows that the proposed approach provides relevant uncertainty diagnosis. Additional results on uncertainty quantification can be found in the supplement.
6 Conclusion
Aczrnoo
In this paper, proposed general Bayesian methodology based on a nat- ural mixed-effects modeling approach to model personalisation in federated learning. Our FL method is the first that allows for both personalisation and cheap uncertainty quantification for (cross-device) federated learning. By in- troducing common prior the local pa- rameters, we tackle the local overfitting problem in the scenario where clients have highly heterogeneous and small data sets. In addition, we have shown that the proposed approach has favorable con- vergence properties. Some limitations of FedPop pave the way for more advanced personalised FL approaches. As an ex- ample, our model does not allow for training heterogeneous architectures across clients because of the introduced common prior, and only satisfy first-order privacy guarantees. Regarding the latter, further works include for instance deriving differentially private versions of our framework.
In-distrbutton MNIST
10
distribdttn FeshionMNNST
Denie
fog
0.00
0.55
0.50
0.75
Figure 4: (right) Calibration on CIFAR-10 for a specific client and (left) OOD analysis with MNIST training & FashionMNIST inference one curve corresponds to one client.
Acknowledgments and Disclosure of Funding
The authors acknowledge the Lagrange Mathematics and Computing Research Center for supporting the project. The development of the algorithm and conducting experiments (Section 5) was supported by Russian Science Foundation grant 20-71-10135.
10

---

References
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication- efficient SGD via gradient quantization and encoding. Advances in Neural Information Processing Systems, 2017.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated Learning with Personalization Layers. arXiv preprint 19Xiv: 2019.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-9, IEEE, 2020.
Thang D. Bui, Cuong V. Nguyen, Siddharth Swaroop, and Richard E. Turner. Partitioned Variational Inference: A unified framework encompassing federated and continual learning. arXiv preprint arXiv:1811.11206 2018
George Casella. An Introduction to Empirical Bayes Data Analysis. The American Statistician, 39 (2):8-877, 1985.
Hong- You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv preprint arXiv:2009.01974, 2020.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting Shared Represen- tations for Personalized Federated Learning. In International Conference Machine Learning. pages 2089-2099, 2021.
Luca Corinzia, Ami Beuret, and Joachim M. Buhmann. Variational Federated Multi-Task Learning. arXiv preprint arXiv: 1906:9906000000 962686 2019.
Arnak S. Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. Journal of the Royal Statistical Society, Series B, 79(3)6511-676, 2017.
Wei Deng, Yi-An Ma, Zhao Song, Qian Zhang, and Guang Lin. On Convergence of Federated Averaging Langevin Dynamics. arXiv preprint arXiv:2112.051120 2021a.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive Personalized Federated Learning, 2021b.
Alain Durmus and Eric Moulines. Nonasymptotic convergence analysis for the unadjusted Langevin algorithm. The Annals of Applied Probability, 27(3):1551-1588 06 2017. doi: 10. 121 1214/16 AAP1238.
Khaoula El Mekkaoui, Diego Mesquita, Paul Blomstedt, and Samuel Kaski. Distributed stochastic gradient MCMC for federated learning. In Conference on Uncertainty in Artificial Intelligence, 2021.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach. In Advances in Neural Information Processing Systems, 2020.
Andrew Gelman and Jennifer Hill. Data Analysis Using Regression and Multilevelftierarchical Models. Cambridge University Press, New York, 2007.
Erin Grant, Chelsea Finn, Sergey Levine. Trevor Darrell, and Thomas Griffiths. Recasting Gradient- Based Meta-Learringg as Hierarchical Bayes. In International Conference on Learning Representa- fions, 2018
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.055/1 05516, 2020.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning, arXiv preprint 22X1v22010.02.37, 2020.

---

Filip Hanzely, Boxin Zhao, and Mladen Kolar. Personalized federated learning: A unified framework and universal optimization techniques. arXiv: 2102.09743, February 2021.
Joey Hong, Branislav Kveton, Manzil Zaheer, and Mohammad Ghavamzadeh. Hierarchical Bayesian Bandits. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, 2022.
Samuel Horváth, Dmitry Kovalev, Konstantin Mishchenko, Sebastian Stich, and Peter Richtárik. Stochastic Distributed Learning with Gradient Quantization and Variance Reduction arXiv preprint arXiv:1904.0511 2019.
Yihan Jiang, Jakub Konevený, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint 1exXiv:9999/ /2488, 2019.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Fichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adria Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He. Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song. Weikang Song. Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Xiong, Zheng Xu, Qiang Yang, Felix Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2) 1-210, 2021. ISSN 1935-8237. doi: 10. 1..15//2220000083.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pages 5132-5143, 2020.
Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar. Adaptive gradient- based meta- learning methods. Advances in Neural Information Processing Systems, 32:5917-5928 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Available http: cifar cifar.html, 2009.
Marc Lavielle. Mixed Effects Models for the Population Approach: Models, Tasks, Methods and Tools. Chapman and Hall/CRC, 2014.
Yann LeCun, Sumit Chopra, Raia Hadsell, Fu Jie Huang, and et al. A tutorial energy-based learning. In Predicting Structured Data. MIT Press, 2006.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Vir- ginia Smith. Federated optimization in heterogeneous networks. In Dhillon, D. Pa- pailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems, vol- ume 2, pages 429-450, 2020. URL https: http:///onnsssmmlll..p/ / 38af86134b65dde
Paper..dd.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust fed- crated learning through personalization. In JCML, pages 6357-6368, 2021. URL http:
pproceedings..r.ppm
Paul Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and Louis Louis-Philippe Philippe Morency. Think locally, act globally: Federated learning with local and global representations. In NeurIPS 2019 Workshop Federated Learning, 2019.
Dongzhu Liu and Osvaldo Simeone. Channel- Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers. IEEE Journal on Selected Areas in Communications, 202
Dongzhu Liu and Osvaldo Simeone. Wireless Federated Langevin Monte Carlo: Repurposing Channel Noise for Bayesian Sampling and Privacy. arXiv preprint 110::2008..06444, 2021b.
12

---

Jeffrey Long. Longitudinal Data Analysis for the Behavioral Sciences Using R. Sage Publications, Inc, 2011.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication--fiiiienn Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 22331288 Fort Lauderdale, FL, USA, 20-22 Apr 2017. PMLR.
Constantin Philippenko and Aymeric Dieuleveut. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees. arXiv preprint arXiv:2006. 1 14591, 2020.
Krishna Pillutla, Kshitiz Malik, Abdelrahman Mohamed, Michael Rabbat, Maziar Sanjabi, and
Lin Xiao. Federated Learning with Partial Model Personalization, 2022. URL https: "/ggerreviee..ne//oormmm' (o-1
Vincent Plassier, Maxime Vono, Alain Durmus, and Eric Moulines. DG-LMC: turn-key and scalable synchronous distributed MCMC algorithm via Langevin Monte Carlo within Gibbs. In International Conference Machine Learning (ICML), 2021.
Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22(3):400-4077 09 1951. doi: 10. //o114////////77795666
Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in statistics, pages 388-394. Springer, 1992.
C. P. Robert. The Bayesian Choice: from decision-theoretic decisson-therreeiccuuutttoooo foundations to computational implemen- tation. Springer, New York, 2 edition, 2001.
Gareth O. Roberts and Richard L. Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341-363 1996.
Karan Singhal, Hakim Sidahmed, Zachary Garrett, Shanshan Wu, John Rush, and Sushant Prakash. Federated reconstruction: Partially local federated learning. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 11220-11232. Cur- ran Associates, Inc., 2021. URL https: / 2c44a200d5aaaaaad222222666222222
///pp//eeding..nurrrrii. httpingwwwe.rrr..pppeeeereeeeeee wttpcc/caapee/22//11100. Paper.pdr.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning, arXiv preprint arXiv: 10467, 2017.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, pages 1-17, 2022. doi: 10.1109/
TINNSS.202.3114. 2022.316069..
Maxime Vono, Vincent Plassier, Alain Durmus, Aymeric Dieuleveut, and Eric Moulines. QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning. In AISTATS, 2022.
Jianyu Wang. Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data. Suhas Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman. Martin Jaggi. Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi, Sashank J. Reddi, Peter Richtarík, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. Field Guide to Federated Optimization. arXiv preprint arXiv:2107.0600 06917, 2021.
13

---

Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics, In International Conference on Machine Learning, 2011.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian -agnosicc meta-learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: :ttppp::://a/ wwroceedings..ouris...o. 2018 000.fff.eeeeee2444........226660 Paper.ppp.rrr..e
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We describe these limitations in the conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set assumptions of all theoretical results? [Yes] Some assumptions are explicitly stated in Section 3. For the sake of readability and due to the space limit, we postponed other technical assumptions in the supplement.
(b) Did you include complete proofs of all theoretical results? [Yes] All our proofs are postponed to the supplement.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (cither in the supplemental material or URL)? [Yes] See the supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] Some details have been stated in Section 5 in the main paper. Because of the limited number of pages, others have been postponed to the supplement.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We did it for some experiments we conducted, see e.g. Figure 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type GPUs, internal cluster, or cloud provider)? [Yes] This is postponed to the supplement.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See for example the citation for CIFAR image data sets.
14

---

(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "kIJ8wWs9ffh", "reviewer": "Reviewer_BRBt", "summary": "This paper proposes a novel method for personalized FL having uncertainty quantification by using the population modeling paradigm where the client's models involve fixed common population parameters and random effects. From their empirical results, their method is robust to client drift, practical for inference, and uncertainty quantification under mild computational and memory overheads. Their method FedPop performs personalization and cheaply works for quantifying the uncertainty in an on-device manner.", "questions": "Q1) During the training dynamics, how and why can FedPop is robust to client drift, practical for inference on new clients?\n\nQ2) Is your theorecial result empirically guaranteed after the FL training? how does the trained model behavior have?\n\nQ3) Do your methods have robustness even in I.I.D. settings?", "limitations": "Despite their significant empirical results, there remain limitations: 1) More other methods should be conducted to verify its robustness in applications to advanced Personalized FL methods. 2) ablation studies on each factor of FedPop such as warm start strategy, gradient estimator etc... as well as hyperparamter settings of other regularization methods such as weight decay, augmentation...", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strength\n- Their claim is supported by both theoretical and empirical results. \n- The authors point out a major problem in related field.\n\n* Weakness\n-  the set-up of experiments and results reported in this paper are still questionable. (Datasets, Architectures, FL settings such as participation ratio, data heterogeneity settings with Dirichlet distribution)\n- The empirical evidence for Theorem 1 may be needed. \n\n\n\n\n------updated after rebuttal---------\n\nThank you for your detailed response for my raised concerns. still I think this work needs more comparisons with related FL approaches based on bayesian approaches, but most concerns are solved. So,  I decided to increase my score from 4 to 5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "asFsE9L3SY6", "reviewer": "Reviewer_8AYd", "summary": "This paper proposes a new algorithm FedPop for personalized federated learning by modeling it as a mixed-effects probabilistic problem. Specifically, a client’s local distribution depends on a fixed-effect part (constant for all clients) and a random-effects part (captures the heterogeneity).  FedPop is developed by maximizing a posteriori probability via stochastic approximation. The algorithm works as follows: in each round, the participating clients will use MCMC to generate samples of the random-effects random variable, use them to compute approximate gradients, and send the updates to the server. A convergence bound is established for the proposed algorithm under appropriate assumptions. The authors empirically show that the proposed algorithm outperforms related works over a synthetic dataset, and two versions of CIFAR datasets (CIFAR-10 and CIFAR-100).", "questions": "See my detailed comments above for questions and suggestions, especially my concerns on application in the large-scale cross-device setting and the missing baseline FedAvg+FT with only fine-tuning the last layer (note: both the fine-tuning learning rate and number of epochs are hyperparameters that should be properly tuned from data).", "limitations": "The authors pointed out the limitations and future work in Section 6.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper is well-written and easy to read. \n\n- Modeling the personalized federated learning problem using the mix-effects paradigm seems new in the literature. The proposed algorithm allows each client to learn a personalized model and at the same time perform uncertainty quantification.\n\n- In addition to the empirical comparisons, the authors provide theoretical guarantees and establish convergence bounds for the proposed algorithm.\n\nWeakness:\n\n- My main concern is that while the paper claims that the proposed algorithm is designed for the cross-device FL setting, it is unclear to me that FedSOUK (Algorithm 1) is indeed suitable for real-world cross-device applications. Cross-device settings typically involve a very large population (e.g., hundreds of millions mobile devices), and it is common that a device only participates at most once during the training process [1]. As a result, cross-device setting usually needs *stateless* algorithms rather than *stateful* algorithms. The main difference is whether the algorithm relies on clients maintaining local states from one round to another (see section 5.1 of [2] for why the stateful algorithm performs poorly in the cross-device setting). The proposed algorithm FedSOUK (Algorithm 1) is a stateful algorithm, because the participating clients warm-start MCMC from the $Z$ value from the previous round. In the real-world cross-device setting where most clients participate only once in training, this means that warm-starting is impossible, will this constraint affect the performance of the proposed algorithm? In that case, will each client need to draw a lot of $Z$ samples locally (how do we set $M$ in practice)? All the experiments shown in the paper are using datasets with a small number of clients (b=100), which is a scale of typical *cross-silo* federated settings [1]. It is thus not convincing to me that the proposed algorithm can perform similarly well in the real-world cross-device setting with millions of model devices and really low sampling rate per round (where most clients participate at most once in the entire training process). \n\n- Other comments/questions:\n  - FedAvg + fine-tuning (FT) a simple, effective, and stateless personalization algorithm (see, e.g., [3][4]) used in real-world cross-device FL applications (e.g., [5]). It is better to include that in Table 2 for a more comprehensive comparison (similar to Table 1 in [6]). On image datasets like CIFAR, fine-tuning only the last layer is usually better than fine-tuning the entire model. Besides, use the last layer as the personalization layer is also consistent with the proposed algorithm FedSOUL. Therefore, it is more convincing to add the baseline experimental results of FedAvg+FT with only fine-tuning the last layer to Table 2 (note: both the fine-tuning learning rate and number of epochs are hyperparameters that should be tuned from data).\n \n  - The posterior distribution $p(\\phi, \\beta | D)$ equation (Line 117) does not contain $p(D)$ on the right-hand side, why is that?\n\n  - In Algorithm 1, the step of server “Send {\\beta_{k+1}, \\phi_{k+1}} to clients belonging to A_{k+1}” happens at the end of every round, why does server not broadcasting the values to the participating clients at the beginning of every round? \n\n[1] Kairouz et al., \"Advances and open problems in federated learning.\" Foundations and Trends in Machine Learning,  (2021): 1-210.\n\n[2] Reddi et al., “Adaptive federated optimization.”, ICLR, 2021.\n\n[3] Wu et al., “Motley: Benchmarking Heterogeneity and Personalization in Federated Learning”, arXiv: 2206.09262, 2022.\n\n[4] Matsuda et al., “An Empirical Study of Personalized Federated Learning”, arXiv:2206.13190, 2022.\n\n[5] Wang et al., \"Federated evaluation of on-device personalization.\" arXiv preprint arXiv:1910.10252, 2019.\n\n[6] Collins et al. \"Exploiting shared representations for personalized federated learning.\" ICML, 2021.\n\n-------------------------update after the author rebuttal-------------------------------\nThe reviewers showed additional experimental (and promising) results in the cross-device setting: https://github.com/anonneurips1/rebuttal_neurips/tree/main\nOne limitation of the proposed method is that it requires each client to maintain $M$ samples each with dim($z$), where $M$ is around 50 in the cross-device setting. If dim($z$) is similar to dim($\\phi$) (which is the case for the StackOverflow experiments), this means that the memory cost is 50 times that of other personalized algorithms such as FedAvg with fine-tuning. \n\nI decided to increase my score from 5 to 6 because: 1) the memory&computation limitation is common for other Bayesian FL algorithms; 2) the proposed algorithm seem a strong baseline when the model size is small (or when the client's local resources are not constrained, such as in the cross-silo setting).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B4mpRCUl0X6", "reviewer": "Reviewer_f916", "summary": "This paper introduces Bayesian paradigm to personalized FL, allowing for both personalization and cheap uncertainty quantification. Specifically, this paper introduces the fixed part (hyperparameter \\phi) to capture the common representation of all local data and the random part via a common prior modeling data heterogeneity. The authors derive the non-asymptotic convergence guarantees for their federated stochastic optimization algorithm.", "questions": "Please see the weaknesses part.", "limitations": "As I can see, this paper has no potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Merits:\n\n1) The paper tackles an important problem in federated learning, which allows uncertainty quantification for personalized FL.\n2) FedPop reduced the memory costs by introducing the mixed-effects instead of high-dimensional hyperparameter.\n3) This paper provides convergence guarantees for the proposed algorithm.\n\nCons:\n1) The novelties of proposed algorithm are unclear to me. It seems the difference between Bayesian FL and FedPop lies in the design of probabilistic hyperparameter. FedPop use two hyperparameters instead of a high-dimensional hyperparameter, including a common prior, to reduce storage costs.\n2) The algorithm uses many independence assumptions, for example the independence on \\phi and \\beta at line 114 and the independence between local likelihood functions below line 117.\n3) The authors may introduce more discussions on the computational complexities and communication burdens. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept