Federated Variational Inference: Towards Improved Personalization and Generalization
Authors: Elahe Vedadi
Joshua V. Dillon
Philip Andrew Mansfield
Karan Singhal
Arash Afkanpour
Warren Richard Morningstar
Venue: Accepted by TMLR ()

Abstract: Conventional federated learning algorithms train a single global model by leveraging all participating clients’ data. However, due to heterogeneity in client generative distributions and predictive models, these approaches may not appropriately approximate the predictive
process, converge to an optimal state, or generalize to new clients. We study personalization and generalization in stateless cross-device federated learning setups assuming heterogeneity in client data distributions and predictive models. We first propose a hierarchical generative model and formalize it using Bayesian Inference. We then approximate this process using Variational Inference to train our model efficiently. We call this algorithm Federated Variational Inference (FedVI). We use PAC-Bayes analysis to provide generalization bounds for FedVI. We evaluate our model on FEMNIST and CIFAR-100 image classification and show that FedVI beats the state-of-the-art on both tasks.

Published in Transactions on Machine Learning Research (08/2024)
Federated Variational Inference: Towards Improved Person- alization and Generalization
Elahe Vedadi Google Research
elabeveladii@goggee.com
Joshua V. Dillon Google Research
judillon&@google...o
Philip Andrew Mansfield Google Research
memes@google.com
Karan Singhal
keronsinghhul@ggoggee.com
Google Research
Arash Afkanpour Google Research
arashaffigoogle.com
Warren Richard Morningstar
wmorning@ggoggee.com
Google Research
Reviewed on OpenReview: https: /ttps://openreee optp//vee..eeeeeeee net/ forum? 60mRkUHgs5
Abstract
Conventional federated learning algorithms train a single global model by leveraging all participating clients' data. However, due heterogeneity in client generative distributions and predictive models, these approaches may not appropriately approximate the predictive optimal state, or generalize to clients. We study personalization and generalization in stateless cross-device federated learning setups assuming heterogeneity in client data distributions and predictive models. We first propose hierarchical generative model and formalize it using Bayesian Inference. We then approximate this process using Variational Inference to train our model efficiently. We call this algorithm Federoted Varia- tional Inference (FedVT). We use PAC-Bayes analysis to provide generalization bounds for Ve We evaluate our model on FEMNIST and CIFAR-100 image classification and show that FedVI beats the state-of-the-ass on both tasks.
process, converge
1 Introduction
Federated Learning (FL) (MeMahan et al. 2017) allows training machine learning models on decentralized datasets, avoiding the need to aggregate data on a central server due privacy concerns. Tn FL,, the central server oversces global model distributed to clients who conduct local training, and the model updates are aggregated to ileratively improve the global model.
In simple and idealized settings, FL can approximate centralized training with similar theoretical guarantees, seen in FedSGD McMahan et al. 2017). However, real-world cross-device FL scenarios, such those in (Reddi et al. 2020 Wang el al. 2021 often diverge from these ideal conditions. Practical FL. implementations involve multiple local training steps to minimize communication overhead. Client participation is typically uneven, with some contributing more data and others not participating at all. Additionally, the non- Independently and Identically Distributed (non-III) nature client datasets, stemming from distinct data generation processes, challenges theoretical guarantees, leads to performance disparities between participating

---

Published in Transactions on Machine Learning Research (08/2024)
and non-particiiating clients Yuan et al. 2022 and complicates training high-performing models in practical FL. setups.
Modern approaches address this challenge by either modifying the local loss to converge to a global solution
ct al. 2020 or using personalized models to handle local distribution shifts (Zhang et al. 2022 Approaches
for personalization have often focused on stateful FI, setups, where clients are revisited throughout training and thus can update a locally stored model (Karimireddy et al., 2020; Wang et al., 2021 However, many production scenarios are effectively stateless, since individual clients only rarely contribute to training, and local models may be either stale or non-existent. Few studies have concentrated on personalization in this context. Those that have (Singhal et al. 2021), require clients to possess labeled examples for personalization.
This paper explores personalization in stateless cross-device FL selups and introduces Federated Variational Inference (FedVI), an algorithm which utilizes Variational Inference (VI) to enable models to generalize and personalize across diverse client data, even for untrained clients. The key contributions encompass (i) proposing hierarchical generative model rooted in mixed effects models for cross-device federated setups, (ii) offering generalization bounds through Probably Approximately Correct (PAC)-Bayes analysis, (iii) introducing FedVI algorithm, inspired by the theoretical approach, which provides simplified experimental approximation and can be implemented by the existing FL frameworks, and (iv) demonstrating the superior performance of VedVT on two federated datasets, FEMNIST and CIFAR-100, compared to previous state-of-the-art methods.
2 Related Work
To tackle statistical heterogeneity in FL, various studies have employed Bayesian methods to incorporate domain knowledge and aid convergence. Early attempts (Thorgeirsson & Gauterin 2020
Bayesian FL:
Chen & Chao 2020) focused on model aggregation, either to retain uncertainty in model parameters, or to weight. parameter updates proportional to performance. Zhang et al. (2022) instead attempts to use Bayesian Neural Network (BNN) approximated with VI to train global model using a Kullback Leibler
(KL) regularizer which induces convergence similar to the proximal term in FedProx (Li et al. 2020). While
their local models can, in principle, personalize by deviating from the global model, they realistical require
stateful settings with significant labeled data on clients in order to do so. KKoteeeskii et al. (2022) casts personalized FL as mixed effects regression, and attempts to model the inherent heterogeneity in this setting explicitly using Stochastic Gradient Langevin Dynamics (Welling & Teh, 2011]. Our proposed method assumes similar generative process to Kotelevskii et al. (2022) but instead uses VT to efficiently infer the posterior, as well place bound on the predictive risk to induce generalization to new clients (Germain
et al.] 2016).
Stateful FL: There is a rich body of literature on personalization in cross device FL (Corinzia et al. 2019 Ghosh et al. 2020 Chen & Chao 2021 Collins et al., 2021 Deng et al.] 2020 Li ct al., 2021 Hassan et al, 2023). Many previous methods Tocus on stateful settings, where local parameters are stored on clients and maintained across training rounds. However, as emphasized in Table 1 of [Kairouz et al., 2021), statelessness is key characteristic of cross-device FL, highlighting its practical significance. Therefore, we focus on stateless settings, where maintaining up-to-date local states on each client is not feasible. This is similar to the setting considered by (Marfoq et al. 2022). ; who uses K-nearest neighbors to account for client distributional shift. While this is a robust means of dealing with both input and output distributional shift, it requires clients to possess labeled examples for every class (which is unrealistic in real-world setups), and cannot be used outside of classification problems. FedVT's statelessness stems from its use of an amortized posterior inference model. This concopt shares similarities with how Variational Autoencoders (VAEs) (Kingma & Welling 2013 perform inference. In VAEs, an encoder network maps input data to a latent space. FedVI takes a set of examples and infers the posterior over the local parameters, effectively capturing the personalized features of each client's data. This dynamic encoding allows for continuous adaptation and personalization as new clients join and contribute their data.
Meta Learning: There is a significant amount of prior work that studies connections between personalized FL and Model-Agnostic Meta-Lcarning (MAML) approaches (Finn et al. 2017: Singhal et al. 2021 Fallah ct al.) 2020 Collins al. 2021 Lin et al., 2020 Chen et al., 20188). The main idea behind these works is to find an initial global shared model that the existing or new clients can adapt to their own dataset.
2

---

Published in Transactions on Machine Learning Research (08/2024)
by performing few steps of gradient descent with respect to their local data. FedRecon Singhal et al. 2 is also motivated by MAMT, and considers partially local federated learning setting, where only a subset of model parameters (known as global parameters) will be aggregated and trained globally for fast roconstruction of the local paramcters. Our work can be considered as an extension of FedRecon. Unlike this work, we also provide a means of reconstructing local parameters I without access to labeled data.
3 Methods
3.1 Hierarchical Generative Model
Let. us consider & stateless cross-device federated setup with multiple clients and a central server, where randomly selected client subsets participate in each training round. In this setup, we categorize each client's model paramcters as global (0) and local (Bx for h E [c[p parameters, with c representing the total number of clients. Global parameters update at the server end after each training round, while local parameters are deleted after each round. Global parameters are drawn from the prior distribution ((O), while each client's 's local parameters are independent samples from the local prior rBB)). Additionally, data may not exhibit IID characteristies among clients, i.e., Iik ~ V& (Xx) for i E [ng] and k E [c], where ny is the total number of data samples at client k. Moreover, each client may have distinct predictive distribution. Although all clients share the same likelihood distribution family (O Be, rax)), the distribution varies based on 8k, making it different for each client.
The above setup is a prototypical example of mixed effects model (Demidenko/ 2013), commonly employed for predicting continuous random variable using multiple independent factors, including both random and fixed, and incorporating repeated measurements from the samne observational unit. Mixed effects models Demidenko 2013) have a well-established foundation. By framing our setup within this context, we can leverage existing theoretical insights in this field. To summarize,, we propose the following hierarchical data generating process:
0 ~ t(0)
(1)
for & E [c]
3x ~ r r(BB)
for [nw] : Fik va (Xx) Ya ((Yr) 3c, axx)),
where f: 1 B& X'- + Z& is deterministic function (e.g., DNN) mapping what we know to the latent. space ZR, which is the parameter space of our distribution over outcomes, ((.).
For more intuitive grasp of varying data generation processes and predictive distributions, consider the Federated EMNIST dataset (FEMNIST; Figure [ where each client's dataset consists of numbers and letters handwritten by that client. Each client's input data reflects their unique writing style; for instance, a German client may include a horizontal middle bar when writing sevens, whereas an American client may not. Likewise, the German client may add a hood to the number 1, while the American client may not. This describes the difference in data generating distributions. This also illustrates that each client may have different predictive distributions: the American client may see the German's may see the American's 1 as a lowercase 'I". Thus their predictive distributions are in direct conflict with each other. A purely global model cannot accommodate this diversity and must incorporate some level of local adjustments to accurately represent the data generation process. Our proposed algorithm explicitly assumes this data generating process. Note that this assumption reduces in special cases to existing FL setups, such as IID predictive distributions (r(B₂) 8(BB - B)), or IID data generating processes (v*(X*) = v(Xx)). In the following section, dctail how we use VI to efficiently infer the model parameters.
7, while the German client
The detailed procedure for reconstructing the local parameters can be found Section 2I this paper we represent. the of (1.....c) by (c].
3

---

Published in Transactions on Machine Learning Research (08/2024)
Inputs Germant German7
Labels
Inputs
Labels
Clienti
German1
Clien2
American1
American7
1
7
1
7
7
7
7
7
7
7
7
7
German1-y1
German7--1(11)
American1-v2(X22
American7-~2(X2)
Figure 1: Illustration of diverse data generation and predictive models in cross-device FL.
3.2 Training Objective
In this section, our goal to present a step-by-step definition the objective function that is meant to be minimized throughout the training process. We begin by calculating the estimated probability density function of labels given input data, denoted as p(ly" def p(ly" "M }H***))), following similar marginalization approach as (Watanabe 2018
P( def 1.12. 1./....
/ p(0, 1","" =" "")(""""****)""""""""""""""""""" (*)))
(2)
where go def { def (3x & e [c]] an del (z. [ne]], y** def (u wi:ie [mx]], fr def {x. i
e
[n [c], and {y def (uw [na], [c].
Therefore, for calculating p p(v**))) it is required to calculate the posterior probability of model parameters given the training data which is equal to:
p(0, 1y" x p(0,3" Br, fyrm }(H24))9)
(3)
pl(y" (1nn)))))
Assuming that the prior distribution of the global parameters, ((0), the prior distribution of the local parameters, r(8k), and the likelihood distribution of each client, E(y [f, Bk, x)+)), are independent we calculate the numerator of Equation
as:
p00,8" 1y" 3 H= }o) p(0, 1Kk, {vin Jaei Jostcl (
t(0) II
r(3k) II
II kE(c]
(f,, xx))
kE(c]
4E(ma)
t(0) II
r(Bk) II [(vx)) (k, (k,
kE(A
+(0)r (11 8r, {xm)9)).
(4)
Moreover, the denominator of Equation can be written as:
p((y 19 1 S....../. ) (((2))))
(5)
Unfortunately this integral not only infeasible to compute, but also mathematically intractable. Conse- quently, this makes the whole posterior intractable.
To address the problem of the intractable posterior distribution, a tractable surrogate distribution, denoted as q0, , 2=x =*)}", is approximated using VI. By formulating a specific lower bound on the marginal distribution known the evidence lower bound (ELBO), the best surrogate distribution be obtained
4

---

Published in Transactions on Machine Learning Research (08/2024)
by minimizing the ELBO (Equation 6p. This minimization process provides the best approximation for the intractable posterior distribution, p(0, 1 ""* (o), under the chosen family of surrogates. The notation DKL represents the KL divergence between two distributions p and q, and detailed derivations of Equation are available in Appendix A
- log (a) - - logg (1 39) + min DEL ("1"", 2())))))))))))
a(0, 81 ( yy" p(0. 3r, byme (2(x)))))) }o)
min
flg
(6)
q
By asserting factorization, we define the surogate as parametric distribution as: q(0, Hy" x )9)
def = (x(Ol(" "* ** ) II
(3,00, ,²*) def qx x"* 19x {y (y" }o),
kE(c]
where À the parameter that uniquely defines the surrogate distribution. Therefore, the objective function for training the proposed hierarchical model is a generalization of the negative ELBO (specifically, negative ELBO when 7 1), which can be written follows using the definition of KL divergence, logarithm propertios, and the multiplication rule in probability.
(ar J(A:Y, = (o 1o) [log q0, 12"" " (9) p(0, Bc, fyn (=
Per Datum Expected Loss
Σ
Rx (10
[- log ElVi [ BR, (0, ...R))
kElel (E(ma]
Thh + y DDK "* }())
Σ kElc
(DLL (B,l0, y"M x )1r(())))) (8)
Global Regularizer
Local Regularizer
where Y, T, t(0), (5x), and the functional form of qu (0, (y** (""**="** are left as hyper parameters. The details of this derivation are provided in Appendix B In the following section we explain how minimizing this objective function is equivalent to minimizing an upper bound the generalization error.
4 Generalization Bounds
As mentioned earlier, we utilize a generalization of the negative ELBO as our objective function to train the hicrarchical model. Minimizing this function ideally reduces the training dataset error (empirical risk). However, our primary aim is minimize the error on unseen datasets (generalization error true risk) for better generalization. To achieve this, conduct PAC-Bayes analysis, leveraging the results presented in Theorem 3 of (Germain et al. 2016). We introduce slightly generalized version of this theorem in the form of the following corollary, cnabling us to compute a generalization bound for the true risk of our model, under the assumption of non-IID empirical data samples.
Corollary 1 Given a distribution D over X' X'y, a hypothesis set F = {0, (8,8"), loss function é: FxXxX) R, a prior distribution r(O,B) (O)r(B)))))))))))))))))))))))() over F, 8 E (0, 1] and a real number n O, with probability at least 8 over the choice of ((x") me) (X,Y) D, for any q(-) on F we have:
True risk
Ep( log (En(0, 0, S
Empirioat risk
KL dinergence
Ex.y Sx.r(Eer0.sxxxxy log log(ee) (005,,,,))1 )I) - (Xx
+ log (fx Ex((,,0)) exp
(9)
Stack term

---

Published in Transactions on Machine Learning Research (08/2024)
Posterior Constructor Model
p( x
Support Latent Features
Support
6'
Sampling
21x"1/xx; p., 01)
Data
Global Query Latent Features
Embedding Model
Global Classifier
Merge fn
fn
Query
fxr: Br. @)
fx
Local Query Latent Features
Local Classifier
Figure 2: Our proposed model architecture implementing VedVI.
T (og(((vvx(zr 0, Bx)) and Ep[.] = Exx,))=[[]
Where (xx]|oo(((XXXx,,,,9))
This corollary's proof closely follows Theorem 3 in Germain et al. 2016): We establish it using Jensen's inequality, Donsker-Varadhan Varadhan change of measure inequality, and Markov's inequality, Additional details can be found in Appendix
Sketch of Proof:
Having obtained the generalization bound in Equation we observe that it equals the negative ELBO (Equation 8) (for 1) plus a constant slack term, unrelated to the surrogate posterior distributions. Note that given Equation validity for any > O, setting _ min(y, allows us to consider Equation g plus a slack term as an upper bound on the True risk. Consequently, as long as this slack term remains fimite, minimizing Equation g with respect to the surrogate distribution is equivalent to minimizing the generalization error with respect to the surrogate distribution. Thus we conclude that, assuming finite slack term and with probability greater than 1 5, minimizing Equation should improve the generalization of our model.
5 Implementation and Experimental Evaluation
The main goal this section is to go through the details of our primary theoretical assumptions and model architecture for implementing an instance of our proposed hierarchical generative model and evaluating it. We note that the model architecture proposed in this section is of infinity many architectures that is compatible with our theoretical model. This section will outline how the algorithm's 's steps correspond to the specific components of the hierarchical model and variational approximation presented in Section
Distributions: For the prior distribution of the local paramcters, we assume a normal distribution with zero mean and variance equal to that given by the initialization scheme (e.g. Glorot & Bengio 2010 Glorot et al. 2011 He et al. 2015). We assume that clients' data generating distributions have the same support, but that they can be different from one another. We use a categorical distribution as our likelihood, where the logits generated by deep neural network parameterized by 0 and 8& (described below). To simplify implementation, we use a point estimate for the global posterior. This is equivalent to assuming the hyper parameter of the global KL divergence is equal to zero, i.e., in Equation g we have y = 0. Moreover, to make sure that the KL divergence between the global posterior and the global prior, Dkr. (" x"" }()(())))) is finite we assume that the global posterior is a very narrow normal distribution, but still finite, while the global prior can be any finite function.
Tasks: We evaluate FedVI algorithm on two different datasets, PEMNIST Caldas al. 2019 (62-class digit. and character classification) and C0FAR-100 Krizhevsky et al. 2009 (100-class classification. FEMNIST
data

---

Published in Transactions on Machine Learning Research (08/2024)
is particularly relevant since it has naturally different data generative distribution for each client. Although CIFAR-100 data is synthetically partitioned using a hierarchical Tatent Dirichlet Allocation (I.DA) process (Li & McCallum 2006) and distributed among clients, we evaluate FedVI on this dataset as well to show the superiority of our method on a more complicated classification task.
Model Architecture: There are infinitely many model architectures which could implement our method. The architecture that we chose in our experiments is illustrated in Figure 2 and summarised in Algorithm !
The mathematical notations that are used in both Figure 2 and Algorithm I are as follows:
D def {x"*,""* thth : xh E Nk. n** E Vk. & E [c]] (input dataset of client k)
(10)
Nx = Rixixj
def
(input space; whitened images)
(11)
8kk def
= [c]
(label space)
(12)
Eg (.): N3 + _2d
(embedding model; relu-convnet with dropout)
(13)
Po
(.) Rdd + I (D-
(posterior constructor model; relu-mlp)
(14)
G
(.) ;) RR40 RR D'
(global classifier; one dense layer)
(15)
LB (-) R.d + RuDu
(local classifier; one dense layer)
(16)
0 0' U 0" 0
(global parameters)
(17)
Bx
(local parameters of client k),
(18)
where for FEMNIST we have i = =288, j 1, and Dir = = 62, and for CTFAR- 32, j 3, and 100, for & [c]. For both datasets the embedding size d 128 and the number of local and global features are equal to di 26 and dy 102, respectively.
D&l = c- =
Our proposed model architecture consists of four separate modules: an embedding model, Ear which encodes the imput as a vector, a posterior reconstruction model, P (.)(.), which predicts the posterior over local parameters, a classifier parameterized by global parameters, Ggrr(.), and a classifier implemented by local parameters, Lg, (.), generated by sampling from the reconstructed posterior. The global parameters serve the purpose of classifying input data samples by considoring their global features shared among all clients. On the other hand, the local parameters play a distinet role in refining the classification outcome by accounting for the unique local features specific to each individual client. Our model follows the stateless definition outlined in Table 1 of (Kairouz et al. 2021), eliminating the necessity to retain prior client states for parameter updates. Clients are not required to store updated global parameters; instead, the server aggregates and transmits averaged updates for upcoming rounds. Furthermore, clients can avoid the need to store updated local parameters by employing the posterior constructor model in each round to reconstruct the local parameter distribution, allowing them to derive local paramcters through sampling from this reconstructod posterior distribution.
Implementation: We implement our FedVI algorithm in TensorFlow Federated (TFF) and scale up the implementation to NVIDIA Tesla V100 GPUs for hyperparameter tuning. For FEMNIST dataset with 3400 clients we consider the first 20 clients as non non-participating users which are held-out in training to better measure generalization as in YYuan et al. 2022). At each round of training we seloct 100 clients uniformly at random without replacement, but with replacement across rounds. For CIFAR-100 with 500 training clients, we set the data of the first 10 clients as held-out data and select 50 clients uniformly randomly at each round. We train FedV1 algorithm on both FEMNIST and CIFAR- 100 for 1500 rounds and each round of training we divide both datasets into mini-batches of 256 data samples and used mini-batch gradient descent algorithm to imiiee the objective function. The training procedure for each client k at round 6, outlined in Algorithm [ is as follows. Further details regarding each step are explained subsequently:
Each client is partitions its input data, DR, over the batch dimension into support and query sets, Dx.s and Da.q: using the data split function, S(.). Similar to FedRecon (Singhal support set is used to reconstruct the local parameters and the query set is used to make predictions.
et al., 2021). the
7

---

Published in Transactions on Machine Learning Research (08/2024)
Support Set
Global Support Latent Features
Data
Embediiing Model
Local Support Latent Features
Figure 3: An illustration of the division of the data into support and query sets, as well the division into global and local features.
Note that the support set we use can be unlabeled, and that the two sets need not be disjoint. However, we use disjoint sets in our experiments since (Singhal al.) 2021) found that it improved their model performance.
2. Both support and query sets fed into the embedding model, Eø (.), extract vector representations of the data, i.e., RR,s Rd and Rg.g R
3. The representation for both support and query sets are further split over their features axis into global and local features, i.e., (Rg, Rd,, Ri E B4) and (Rg .9 IRo, R! E Rd),, for d dg + de, using the feature split function F(), as illustrated in Figure 3
4. The global features of the support set, Rig R4,, are used to reconstruet the mean and variance of the local posterior, i.e., (a* E Rd:Diil 0h E IBde)) through the posterior constructor model, Po (.). The local parameters, ( are generated by sampling from this posterior.
The global features of the query set, R2, E RR,, are passed to the global classifier, Gø (.) to get. the global predictions, 0% E RD',, and the local features of the query set, Ry,, Rd,, and local parameters, ((") are passed to the local classifier, Lgz (.), to get the local modifications to the global predictions, Og E TR(Ditl.
6. The local and global predictions are merged computed between these predictions and labels and added to the KL divergence between local posterior and prior.
F to get the predictions. The log-likelihood is then
7. Global parameters get updated through back propagation over the loss function that is calculated in the previous step. This indirectly updates the local parameters, as they are inferred from the posterior constructor model, which is parameterized by the global parameters. Then the local update of the global parameters, (t) along with the number of query data samples at client k n,, are returned to the server.
8. The server aggregates all client updates and calculates the global update of the global parameters, ((++1), and shares them with all clients k E s((+1) for the next round of training.
Data Partitioning: First we note that for both FEMNIST and CIFAR-100 datasets, at each epoch we consider the first 50% of each mini-batch as the support set and the other 50% as the query set (i.e, for a mini-batch with 256 data samples the first 128 samples belong to the support set and the rest belong to query set). For the global-local features split, we found that using a larger number of global features (80%) than local features (20%) performed best. More specifically, these experiments that the dimension of the last, layer of the embedding model is equal to d 128, the first. 102 features are considered as the global features and the rest of 26 features are local features.
Embedding Model: In our experiments the embedding model, (.), is a relu convnet. For FEMNIST experiment we consider the convolutional model with 2 convolution layers that is described in Table of
While there are multiple ways that one could merge the predictions, we found that the simplest way was to add them together. This treats the local predictions as modifications the global predictions in Logit space.
8

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 1
VedVI Training
Input: set of global parameters 8, data split function S(.,, feature split function F(.), embedding model
(g)()); posterior constructor model Par (.), global classifier Gø local classifier Ls, (-), merge function
f(), client update algorithm (.).
ClientUpdate::
(Dk,s (kgg) t S(Ds)
Ra,s < (x pri))
Server Executes:
Ru.g + Eg (I )((t)
g + (imitialize 0)
(R Rkas)) + F(
for cach round t do
(R% Rk + F(Bks.g
800) + (randomlyy sample & clients)
(10R, σk) + Por(RR Q2()))
for each client k E St0) in parallel do (A nk) + Client o(9)
86 + sample(NV(ss ok))
0g t G (R Bu)(t)
end for
n = Ekesto nk
O + Lis (1lk 2
0((+1) - Q(0) ag ExEsiO Tt A (1)
00 + U(f(O)) Oh), Of).yamn )
end for
A < 0 -((1)
Thk DDx
return (A (t)
m&) to the server
Reddi et al. (2020)) paper (without the top layer) and is parameteraized by the global parameters. the detailed structure of this embedding model is as the following.
For FEMNIST: E& (.) conw(32) + retu + conv(64) + relu + mazpool(2,2) + dropout(0.25) + flatten + dense(128) + dropout(0.5)
We choose a convolutional embedding model for CIFAR-100 as well, which similar to FEMNIST embedding model, but having convolution layers instead. The detailed structure as follows.
For CIFAR-100: (-) conv(32) > relu conv(64) - relu cono(128) -> relu cone(256) relu conu(512) + relu + (2, dropout - flatten dense(128) + dropout(0.5)
Posterior Constructor Model: The posterior constructor model, (.),,), is an MLP with three (dense) layers that takes the global features of the output of Egr as imput and generates mean, variance, and bias of the posterior.
For both FEMNIST and CIFAR-100: (.) dense(256) + relu dense(256) + relu dense((2 26 1) (D&))
Global and Local Classifiers: For both FEMNIST and CIFAR-100 experiments global classifier is one dense layer with 11 units and no activation function, parameterized by the global parameters, and the local classifier is one dense layer similar to the global classifier, but parameterized by the local parameters.
Optimizers: We use Stochastic Gradient Descent. (SGD) for our client optimizer and SGD with momentum for the server optimizer for all experiments (Reddi et al., 2020). We set the client learning rate equal to 0.03 for CIFAR-100 and 0.02 for FEMNIST dataset, and server learning rate equal to 3.0 with momentum 0.9 for both FEMNIST and CIFAR-100 datasets.
Evaluation Results and Discussion:
We evaluate our proposed FedVI algorithm against state-of-tho-art
personalized FL method, KNN-Per (Marfoq el al. 2022), as well as FedPA (Al-Shedival el al. 2020), FedEP Guo et al. 2023 (using highest reported values), FedAvg++ CCen & Chao 2021), ClusteredFL (Ghosh et al. 2020), DITTO (Li al., )021), FedRep (Collins et al. 2021). APFL (Deng et al.) 2020), and FodAvg
(MeMahan al. 2017). Results for the bascline methods (except for FedPA and FedEP) are taken from Marfoq et al. 2022).
9

---

Published in Transactions on Machine Learning Research (08/2024)
Table Test accuracy the participating/non het//oon-peee pow-patiiciipatingggggggg clients. FedVI results are reported 7 10-9 for FEMNIST, and T 10-3 for CIFAR-100.
Dataset
FedAvg
FedAvg+
ClusteredFT.
DITTO
FedRep
APFL
FedPA
FedEP
KNN Per
FedVT
FEMNIST 83.4/83.1
84.3/84.2
83.7/83.2
84.3/83.9
85.3/85.4
84.1/84.2
87.3/NA
86.6/NA
88.2/88.1
90.3
CIFAR-100 47.4/47.1
51.4/50.8
47.2/47.11
52.0/52.1
53.2/53.5
61.7/49.1
46.3/NA
50.7/NA
55.0/56.1
59.1/58.7
KNN-Per MMarfoq al. 2022 achieves personalized federated learning by combining global MobileNet-V V2
model with local k-nearest neighbors models based shared data representations, demonstrating improved accuracy and fairness compared to other methods. FedPA (Al-Shedivat et al. 2020) reimagines federated learning as posterior inference problem, proposing a novel algorithm that utilizes MCMC for efficient local inference and communication, employing CNN models for FEMNIST and ResNet-18 for CIFAR-100, FedEP (Guo al. 2023) similarly reformulates federated learning as a. variational inference problem, using an expectation propagation algorithm to refine approximations the global postcrior, and also employs CNN models for FEMNIST and ResNet-18 for CIPAR-100. APPL (Deng et al. 2020 offers communication- efficient federated learning algorithm that adaptively combines local and global models to learn personalized models, utilizing various models (logistic regression, CNN, MLP) on diverse datasets, including FEMNIST and CIFAR-100. ClusteredFL, (Ghosh et al. 2020 is designed for clustered users, iteratively estimating user clusters and optimizing their model parameters, demonstrating strong performance various settings. Finally, FedRep (Collins et al.) 2021) learns a shared representation and unique local heads for each client, using 5-layer CNÑs for CIFAR and 2-layer MLP for FEMNIST, while DITTO {Li et al. 2021 introduces a simple personalization mechanism within multi-task learning framework, utilizing CNNs, logistic regression, and linear SVMs for different datasets.
Test Test Accurady of Holdout Clients
0.8
0.6
0.4
FEMNIST (T= 1
'CIFAR-100 (=
500
1000
1500
Number of Rounds
Figure 4: Non-participating test accuracy of FEMNIST (r= 10-9) and CIFAR-100 (7 10-3) over 1500
rounds of training.
The performance of FedVI algorithm and other methods on the local test dataset of each client (unseen data at training) are provided in Table for participating and non-participating (completely unseen during training) clients. All of the reported values are average weighted accuracy with weights proportional to local dataset sizes. To ensure the robustness of our reported results for FedVI, we average test accuracy across the last 100 rounds of training. Figure illustrates the non-participating test accuracy on FEMNIST ( ( 10-9 ) and CIFAR-100 (7 10-3) over 1500 rounds of training, providing visual representation of the results reported in Table
Figure 5a shows the average test accuracy over the last 100 FEMNIST training rounds for a range of KL hyperparameter т, from 0 to 10. Notably, = 10-9 outperforms others, achieving higher accuracy with a smaller generalization gap compared to t = 0. Appendix D provides further insights and analysis regarding this experiment.
Figure 5b displays the average test accuracy over the last 100 rounds in CIFAR-100, with varying KL hyperparameter Notably, 10-3 achieves the highest accuracy for both participating and non-
10

---

Published in Transactions on Machine Learning Research (08/2024)
participating clients. Comparing to other valuees (T + 0) reveals that minimizing KL divergence reduces the gap in participation test. accuracy, as anticipated. Note that our objective function in Equation recovers the standard ELBO. Setting T 0
S is
indeed a generalization of the ELBO, where setting T = effectively removes the KL divergence term, resulting in maximum likelihood estimation (MLE). While MLE can sometimes lead overfitting, the KL divergence term the ELBO acts as a regularizer, promoling better generalization. Therefore, values of > O, which maintain the KL divergence term, generally exhibit superior performance and result is smaller generalization gaps. Furthermore, comparing this figure Figure 5a it's evident that the difference in test accuracy between is significantly larger than the difference between 7 0 and т 10-3 in the CIFAR-106 experiment. This suggests that minimizing KL divergence is more critical for FEMNIST than for CIFAR-100. One possible explanation ion is that in FEMNIST, each client's 's data generation distribution naturally differs, while in CIFAR-100, data is synthetically partitioned and distributed among clients.
and T 10 in the FEMNIST experiment
Accuracy
CIFAR- 100 Teet Accuracy
0.54
0.52
0.86
0.5
Hold out clients
out
0.85
Participating clients
0.48
Participating
0.84
0.46
10
10
Hyper Parameter (v)
10*
10
KL Hyper Parameter (v)
10
10
10
10
(a) FEMNIST
(b) CIFAR-100
Figure 5: Participating and non non- pon-aatiiiiatting test accuracy vs. KL hyperparameter T.
6 Conclusion and Future Work
This work addresses personalization in stateless cross-device federated setups through the introduction of FedVT, a novel algorithm grounded in mixed effects models and trained using VT. We establish generalization bounds for FedVI through PAC-Bayes analysis, present a novel architecture, and implement it, Evaluation on FEMNIST and CIFAR-100 datasets demonstrates that FedVI outperforms stale-of-the-art methods both cases. It is worth noting that in this paper, we employed narrow normal distribution as the posterior for global parameters. However, in future rescarch, intend to explore euhance the modeling capabilities. Additionally, the model architecture presented in Figure ? is just one of several possible architectures that align with our theoretical hierarchical model. In upcoming work we will focus on refining these architectures to optimize performance and explore their potential for achieving even better results.
generalized distributions to
References
Maruan Al-Shedivat, Jennifer Gillenwater, Eric Xing, and Afshin Rostamizadeh. Federated learning via posterior averaging: new perspective and practical algorithms. arXiv preprint arXtv:2010.0527 2020.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings, 2019.
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated letareannin with fast convergence and efficient communication. arXiv: Learning, 2018. Ctllpttpps//ppp..eennticcccoootr www.rorr//roIiiaa.oooooo
209376818
Hong-Yor You Chen and Wei-Lan Chao. Fedbe: Making Bayesian model ensemble applicable federated Icarning. In International Conference on Learning Representations, 2020. URL Sectps://apt semanticscholar brg/CorpuslD::23611688
11

---

Published in Transactions on Machine Learning Research (08/2024)
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. arxiv preprint or orXim:2107.0077 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pp. 2089-2091 PMLR, 2021.
Luca Corinzia, Ami Beuret, and Joachim M Buhmann. Variational federated multi-task learning. arXiv preprinl Xiv: 06208,, 2019.
Eugene Demidenko. Mized models: theory and applications with R. John Wiley & Sons, 2013.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXic:2003.13401 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. CoRR, abs/ abs/2002.07948,, 2020. URL (ttts://arxxiv.org//b//////
Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126 1135. PMLR, 2017.
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien, PAC- PAC-Baycsian theory meets Bayesian inference. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https //proceedings.nee nww.gss.....pp....
pdf
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586-19597, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks In International Conference on Artificial Intelligence and Statistics, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In GeofTrey Gordon, David Dunson, and Miroslav Dudik (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 315 323, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https:// mlr v15/glorotila.tte
mlr mwr..prsss//
Han Guo, Philip Greengard, Hongyi Wang, Andrew Gelman, Yoon Kim, and Eric P Xing. Federated learning variational inference: A scalable expectation propagation approach. arXiv preprint arXiv:2302.0420 2023.
Conor Hassan, Robert Salomone, and Kerrie Mengersen. Federated variational inference methods for structured latent variable models. arXiv preprint drXiv:2302.03314 2023.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun, Delving deep into rectifiers Surpassing human- performance on imagenet classification. 2015 IEEE International Conference Computer Vision (ICCV), pp. 1026 1034, 2015.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, el. al. Advances and open problems in federated learning. Foundations and Trendsé in Machine Learning, 14(1-2):1-211 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Inter- national conference on machine learning, pp. 5132 5143. PMLR, 2020.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes, 2013. URL https://arxiv..
abs/
12

---

Published in Transactions on Machine Learning Research (08/2024)
Nikita Kotelevskii, Maxime Vono, Alain Durmus, and Eric Moulines. Fedpop: A Bayesian approach for personalised federated learning. Advances in Newral Information Processing Systems, 35:8687- 8701, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 450, 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pp. 6357-6368. PMT.R, 2021.
Wei and Andrew McCallum. Pachinko allocation: Dag-structured mixture models of topic correlations. Proceedings of the 23rd international conference on Machine learning, pp. 577 584, 2006.
Yujie Lin, Pengje Ren, Zhumin Chen, Zhaochun Ren, Dongxiao Yu, Jun Ma, Maarten de Rijke, and Xiuzhen Cheng, Meta matrix factorization for federated rating predictions. Proceedings of the 43rd International ACM SIGIR Conference on Reseurch and Development in Information Retrieval, pp. 2$1-990, 2020.
Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Lactitia Kameni. Personalized federated learning through local memorization. In International Conference on Machine Learning, pp- 2022.
PMLR,,
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273 1282. PMLR, 2017.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H. Brendan McMahan. Adaptive federated optimization, 2020. URL https://arxiv.org
abs/2003. 00295
Karan Singhal, Hakim Sidahmed, Zachary Garrett, Shanshan Wu, John Rush, and Sushant Prakash. Federated reconstruction: Partially local federated learning. Advances in Neural Information Processing Systems, 34: 11220-11232, 2021.
Adam Thor Thorgeirsson and Frank Gauterin. Probabilistic predictions with federated learning. Entropy, (1):41, 2020.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint 21716:2107.06917, 2021.
Sumio Watanabe. Mathematical theory of Bayesion statistics. CRC Press, 2018.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681 688, 2011.
Honglin Yuan, Warren Richard Morningstar, Lin Ning, and Karan Singhal. What do we mean by generalization federated learning? International Conference on Learning Representations, 2022. URL https
//ppenreiiew. www..eeoouunddd-iiieeeeeee
Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational Bayesian inference. In International Conference on Machine Learning, pp. 00223326331 PMLR, 2022.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Appendices
A Derivations of Equation 6
Here we provide the detailed dcrivations of Equation i which are derived basod on Scction 2.2 of (Kiggma
Welling 2013). The main goal of these derivations to devise an upper bound on the negative logarithm of the intractable denominator of the posterior probability of model parameters, i.e,, p(0, """"", (x")")) p(0, 8 (v"* "=""" (9)p((y"* H=**** "*) For this purpose, we consider an arbitrary distribution "y"" }9) as surrogate for the posterior. Since the KT, divergence of two distributions is always non-negative, we can use the KT. divergence between the true posterior and our surrogate to devise an obvious and trivial upper bound on log p( {v"* }"H"" }9) as the initial step in Equation 19 As the minimum of a non-negative number is always non-negative, we replace the KL divergence with its minimum value with respect to the surrogate distribution q(0, Ky ( x""h }"), to make this upper bound as tight possible (Equation 20 Moreover, since D(g((("" } independent of the surrogate distribution, move this term inside the minimum shown in Equation 21
to be able to approximate p(0, (y""" """""""""" ,," }9), in a tractable way.
[ (x=" jo) is
The rest of the proof comes from the definition of KL divergence, the multiplication rule of probability, and
def properties of logarithms. For the sake of simplicity in notation we have f x X, Y in the following equations.
Always 2
- loggp(Y|X) < -logp(Y X) (K x Y ()l (1), Y))
(19)
Always 2
+ - >o)g < - log X) + min DRL (0, 8 X 1) 991,Y))
(20)
-ooggpY((X) S min logpoogp((y)))) (g 3 (p ()1
(21)
(00,39X,, X, Y)
min R [0,00 -aceiixnnllloopP0001 log + log p(0, 30 X Y)
((0, 331X
min E _0 1log
p00,Be|X Bc|, ())(X
g(0, |
min E 1x,)
(22)
p(0, 80, YIX)
B Derivations of Equation 8
We provide details for Equation g which is derived based on the definition of KL divergence, propertics of
logarithms, and the multiplication rule of probability. In the following equations "y"*,""*" def X, Y for the
simplicity in notations.
p(0, 8 YIX)
39, YIX)
200,89X,
p(0, (,Y)
p(0, (X,
g(0, (0,,3
85 YIX)
q(0, (X,
y10,35|X,
((0))((9)((Y115000,
g(0, ((0,89|1 x(,))
(A(X), Y) (3=10,0,,
p(0, (10,8000,,,1
-log(p(Y(X))) log(L(Y)) (x)))
(0 Y), + log(
ga (3990, (1390,X.Y X,Y),
00,,39X,
+ +og((
log( p 80 X,
t(0)
r(3e)
14

---

Published in Transactions on Machine Learning Research (08/2024)
B (x0088-xxx) [log( (x(xX,, }]
t(0)
+ 1,0000111, 1x 1x(00 /log(
2(0, 11,881X,, Y)
)
r(39)
p(0, 8 X, Y)
log(o(((1X)) + (g0, |,,, (1xx,))11(000000xX..
= Ex(0, (- log(e(Y) ;x))]
(((()))) - - B.Acex.y//DKLLuu//00... ()(l((89)) + ( ( (
Expected Loss
-Ey0,.0 log log(e(Y)5(0,3)) X))) + +DKL(xx(OXX, Y) (()))
E (x(()) x) (X
Global Regularizer
Local Regularizer
= =I 20 log( (ogg)/l x))] + (K X X (()(()))
(23)
C Proof of Corollary 1
The proof of this corollary is derived from the proof of Theorem 3 in Germaii et al. (0166)) More specifically,
Equation 24 comes from Jensen inequality, Equation 25 is a result of Donsker- Varadhan change of measure inequality, and Equation 26 comes from Markov's s inequality.
nB log ([(Y]x)) = E)pl log 0,0 $ ( log (e(Y))
(X
(24)
S Yyy (0 log (( 8, [I]]
+ DKL DgI(((0, X, (1)
+ log
Ex(0,,0) (exp 10BD1_10 - -
(25)
$ w.p > 1-8 6 [Ex log ([(Y) 0, DKL 3 - log (!( 19)1
(0,
+ log ( Exx,yEm(o.sv exp ( 8-) //..x/xxx/..//.
(26)
We note that as opposed to Theorem 3 in Germain el al. 2016 we did not assume the empirical data samples (X,Y) are derived IID from a data distribution and interestingly this proof, which a slightly revised version of the proof of Theorem in (Germain al.) 2016), is correct for non-lII empirical data samples well. The rationale behind this is that none of the steps in the aforementioned proof relies on the IID) property of the empirical data samples. More specifically, this proof starts with calculating the true risk, ED, and moving the logarithm inside the expected value using Jensen inequality. After that we use the Donsker- Varadhaan inequality which says [,() < (gl.(gl + (Ex[EE) (Germain et al. 2016). To use this inequality we define &(f) = Ep - Ex.y. The crucial aspect this proof is the Donsker- Varadhan incquality, which holds true for any function &(f) Ep Ex,y and whether the data we used to compute the empirical risk, Ex,y, IID or not,, doesn't. affect its validity. Finally, the last inequality is the Markov's inequality that does not need IID assumption well.
D Extended FEMNIST Experiment Analysis
In this appendix, we further analyze the FEMNIST experiment results establish the robustness of the findings presented in Figure 5a We demonstrate that local regularization consistently improves generalization
Throoghont this paper ((YIX) stands for marginalized likellhood over the surrogate posterior, therefore we have: E(Y|X) = 1x (,,,,,,,XXXXXXX,,,,,,, (,)))
15

---

Published in Transactions on Machine Learning Research (08/2024)
1+9
90
FEMNIST Test Accuracy
89
88
87
86
85
out clients
84
Participating clients
10-11
10
10
10t3
10"3
10"1
10:
KL Hyper Parameter (T)
Figure 6: Average FEMNIST test accuracy for different kl hyper parameters (r), averaged across 5 runs with different random seeds.
90.0
FEMNIST Test Accuracy
87.5
85.0
82.5
80.0
77.5
75.0
Hold out clients
Participating clients
72.5
10²
10a
Number of Hold Out Clients
Figure 7: Test accuracy of FEMNIST (r= 10-9) vs. number of hold out clients.
performance across different experimental sctups, reducing the gap between non-participating and participating clients.
We conducted additional experiments, running the FEMNIST scenario five times with different seed values (0, 10, 20, 30, 40). We then calculated the average and variance of both hold-out and participating test accuracy across these runs.
The results, presented in Figure i provide smoother visualization of the trend. As the figure demonstrates, we still observe a noticeably larger generalization gap for = 0 compare to 7 > 0. This supports our initial findings and further emphasizes the importance of the KL divergence term in promoting generalization, even with increased experimental rigor.
The experiment results provided in both Figure 5a and Figure 0 involved holding out only 20 clients (0.58% of the population) to ensure a fair comparison with other baselines. This small held-out set, might not fully represent the overall data distribution duc to the inherent stochasticity in the remaining 3380 participating clients. To investigate this further, we conducted additional experiments on FEMNIST for = 10-9 with varying sizes of held-out sets (170, 340, 680, 850, and 1700 clients, representing 5%, 10%, 20%, 25%, and 50% of the population). As shown in Figure 7 these results indicate that increasing the size of the held-out set leads to an increase in the participation gap, with the participating clients eventually outperforming the nonparticipating ones. This suggests that the initial observation indeed influenced by the small size of the held-out set and its potential deviation from the overall data distribution.
16


Reviews:
[{"review_id": "LMCrm1iECE", "reviewer": "Reviewer_WAV2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and well-motivated in general. Improving the effectiveness of FL is a promising research direction.\n- The presented idea behind FedVI is easy to follow.\n- Both theoretical and experimental results are presented to justify the method.\n\nWeaknesses:\n- It is not clear about the extra computation and communication incurred by FedVI. By looking at the major algorithm box, FedVI seems to be much more complex compared to FedAvg.\n- The extra computation and communication costs have not been explicitly studied in the experiments.\n- The experimental scales are too small. It would be essential to demonstrate the effectiveness of FedVI on larger-scale datasets, e.g., ImageNet.\n- Some similar baseline methods are missing, e.g., [1-2].\n\n[1] https://arxiv.org/abs/2010.05273\n[2] https://arxiv.org/abs/2302.04228", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Federated Variational Inference (FedVI), a novel algorithm designed specifically to enhance personalization and generalization in federated learning (FL). The authors claim that FedVI more effectively handles data heterogeneity. The FL process is formulated using a hierarchical generative model and formalized using Bayesian Inference. Variational Inference is then used to enhance learning efficiency. Theoretical analyses are provided to justify the performance of FedVI. Empirical studies demonstrate that FedVI outperforms state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As briefly mentioned in the \"weakness\" section, it would be essential for the author to make the following modifications:\n- Adding experiments on the extra computation and communication overheads of FedVI. And study the end-to-end wall clock time comparisons between FedVI and other baseline methods.\n- The experimental scales are too small. It would be essential to demonstrate the effectiveness of FedVI on larger-scale datasets, e.g., ImageNet.\n- Add comparisons between FedVI and baseline methods like [1-2].\n\n[1] https://arxiv.org/abs/2010.05273\n[2] https://arxiv.org/abs/2302.04228", "broader_impact_concerns": "I don't find any Broader Impact concerns in the current manuscript.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NV2PhjHXoF", "reviewer": "Reviewer_QSyx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\nI believe the motivation, both the personalization and the statelessness, for the work is well justified and something that would definitely be of interest for the TMLR audience. Also the connection between you generalization bound in eq. (9) and the ELBO itself is an interesting realization. The empirical evaluation seems to suggest that the proposed model outperforms the existing approach, even with some margin. \n\n# Weaknesses:\n### Writing\nThe writing of the paper needs to improve quite significantly. I do not quite understand for example how the scores computed by the local and global model are \"merged\" before updating the model. This would be a crucial bit of information for understanding how the local predictions can help in personalization. \n\n### Experiments:\nSo do I understand correctly, that if you set $\\tau=0$ in your experiments, then you don't get any regularization from the local priors? If so, I guess the proposed setting wouldn't differ from having a single global model (assuming $\\tau=0$)? Now, looking at Fig. 5a, while there are some points where the generalization gap is smaller than with $\\tau=0$, it seems to be quite hard to predict how different $\\tau > 0$ values work. Would it be possible to somehow repeat the inference multiple times to get less noisy estimates on how the test errors behave as a function of $\\tau$? I'm mainly worried that since the FEMNIST is most likely the data set that has some strong heterogeneity among the clients, and if we cannot clearly witness the regularization there, then it might be hard to say if the regularization actually has some statistically significant effect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an algorithm for learning a variational approximation for local posteriors in federated deep learning task. The motivation for the work is that due to heterogeneity among the clients, learning a single global model might lead to poor generalization. Authors also consider a stateless setting, where we cannot assume that clients can retain the local model (nor the local model) at their device.\n\nInstead of having a single global model, authors propose a hierarchical model for the data generating process over the clients. Using existing literature, authors are able to prove a PAC Bayes type of result. Their Corollary essentially says, that with probability $>1-\\delta$, learning a VB approximation for the proposed model, reduces the the risk over the true data generating distribution (where the slack term of the risk is dependent on the $1/\\delta$).\n\nOn the implementation side, authors learn three separate (global) models: (1) the embedding model, which maps images into a latent space, (2) a global prediction model, and (3) a posterior construction model that outputs parameters for clients variational posterior of the local predictive model. The scores from the models (2) and (3) are combined before computing the prediction loss. The posterior reconstruction model (2), allows the stateless setting. I think the posterior reconstruction model works a lot like the encoding part of a VAE, which maps each data point to the particular posterior means and stds. In the setting of this paper, the posterior parameters are obtained for a batch of client's data.\n\nThe performance of the proposed method is compared against various FL approaches on two data sets FEMNIST and CIFAR-100. On both of these data sets, the proposed algorithm beats the other methods with somewhat significant margin.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major**\n- Further experiments on whether $\\tau=0$ for the FEMNIST leads to significantly larger generalization gap than $\\tau > 0$.\n- Clarify how the merging of local and global predictions is done\n\n**Minor**\n- What $\\tau$ value was used for the results in Table 1 and Fig. 4?\n- Page 4: \"the best approximation\", in general this is not true since the quality of the approximation is completely dependent on the chosen surrogate distribution.\n- Page 4: The formula in the end: I guess this is actually negative ELBO?\n- Corollary 1: Please clarify what is the \"slight generalization\" over the Germain et al. 2016. To me, the bound in equation (9) looks very much the same as the bound in the prior work.\n- About the local parameters. On the 7th point of the listing in page 7, you say that \"Both local and global parameters get updated through back propagation ...\". So I guess as local parameters you mean the $\\mu_k$ and $\\sigma_k$, and not the $\\beta$ (which is titled as local parameters on eq (18))?\n- When you say that \"The local and global predictions are merged to get the predictions\", how is the merging done? The samples fed to both classifiers are the same (just different parts of the feature vector) right?\n- I guess Figure 4 is not referred anywhere in the main text?\n- About Figure 5: If I have understood correctly, the $\\tau$ parameter controls the level of local regularization through scaling the KL divergence between the variational posterior and the prior for $\\beta_k$ parameter. Could you clarify, would the $\\tau=1$ correspond to the actual ELBO objective? Or is there some scaling issue between the data and the prior that might lead to $\\tau \\neq 1$ being the correct scale? If not, then it would be interesting to have more discussion why the test performance gap seems to somewhat high for the $\\tau=1$ in the FEMNIST case, as I would imagine it should lead to better generalization that the $\\tau < 1$.\n\n- Appendix C: in the first line of eq. (24), it seems like you are writing $\\ell(Y \\mid X) = E_{q(\\theta, \\beta^c \\mid X, Y)} [\\ell(Y \\mid X, \\theta, \\beta^c)]$. I assume that the $\\ell(Y \\mid X)$ and $\\ell(Y \\mid X, \\theta, \\beta^c)$ are the evidence and likelihood. Is this really true for variational $q$? Wouldn't you have that $\\ell(Y \\mid X) = E_{\\pi(\\theta, \\beta^c)} [\\ell(Y \\mid X, \\theta, \\beta^c)]$? Or if $q$ is not the variational approximation, can you please clarify what it is?\n- Appendix C: what is the \"||\" after the 2nd $\\leq$ (close to equation 24)? Is it supposed to be KL?\n- Appendix C: I believe the last expectation in the Donsker Varadhan inequality should have a log around it (talking about the expression after \"After that we use the Donsker-Varadhan inequality which says\").\n\n**Very minor**\n- Page 14: \"Morkov's inequality\"\n- Some of the references look a bit odd. There are a lot of uncapitalized letters, as well as missing venues.\n- Possibly relevant cite: Han Guo, Philip Greengard, Hongyi Wang, Andrew Gelman, Yoon Kim, Eric P. Xing: Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach. ICLR 2023", "broader_impact_concerns": "I see no immediate unaddressed broader impact concerns in this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pPsyDdlCiv", "reviewer": "Reviewer_BQTq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It feels as though the paper is split into two distinct parts. In the first, the authors motivate the use of a hierarchical probabilistic model to handle data heterogeneity across clients, and VI for handling the challenge of inference. The generalisation bounds section is a nice addition; however, my main concern is in the correct handling of the hyper parameters $\\gamma$ and $\\tau$. For anything other than $\\gamma, \\tau = 1$, (8) does not lower-bound the evidence, so is not an ELBO as described. The authors should make this clear. Further, I suspect that Corollary 1 is only valid for $\\gamma$ = $\\tau$ also? I could be incorrect on this—clarification from the authors would be appreciated. In the second section, the authors describe their FedVI algorithm. Although one can piece together the parts of this algorithm to relate it back to the hierarchical model and variational approximation described in Section 3, this connection is far from explicit. \n\nSome examples of things that need to be made clearer: \n\n(1) the likelihood is parameterised by combining the outputs of two different functions, one using global and the other using local parameters. (a) make this clear and (b) how does this combination happen?.\n \n(2) The approximate posterior over local parameters, which itself should be clearly defined, is amortised given a subset of the observed data (it would be nice for connections to be drawn to other methods that use amortised inference, such as VAEs). (a) If this is what makes it “stateless”, then this should be made clear. (b) Also, if this is the key contribution, which I think it is, then highlight it. (c) I’m not convinced that this is actually doing VI (i.e. targeting the ELBO), as the expectation in (8) is now over the approximate posterior given the support set and the likelihood is only evaluated at a non overlapping query set. This feels closer to the neural process ML objective (see e.g. [1]). (3) How the posterior constructor model operates as a set function on the set of global features of the query set. Something that would help here is clearly defining the space in which the intermediate variables (e.g. $R^g_{k, q}$) exist.\n\n(3) I understand the notion of “stateless” to mean “no local parameters”. It would be great as to why this is beneficial in this case. Citations to use cases in the introduction would help, and clarity on why it’s an issue to store local parameters vs. computing them using a global model (aren’t these then stored locally anyway to obtain the local predictions?). I’m also somewhat confused by “local parameters remain on clients” in section 3.1—isn’t this exactly what you’re trying to avoid by being stateless?\n\n(4) There is no description of the baseline methods. Are the architectures used the same as the global model? At the bare minimum these should be included in the appendix. \n\n(5) “No assumptions are made about clients’ data generating distributions”. I’m not sure what you mean by this, as you’re using a very specific CNN architecture and categorical distribution which indicates that you are making assumptions.\n\n\n[1] Foong et al. (2020): Meta-learning stationary stochastic process prediction with convolutional neural processes, section 3.2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper frames federated learning as a hierarchical probabilistic model, the key advantage of which being the ability to handle data heterogeneity. As inference is intractable, variational inference (VI) is proposed. The authors develop an approach in which point estimates of global parameters are maintained, and used to amortise inference over local parameters, the result being that local parameters do not have to be stored on individual clients. The method is evaluated on FEMNIST and CIFAR-100.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the issues I have raised above, all of which I believe are critical to securing my recommendation for acceptance.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The contribution and results of the paper are in principle sufficient for TMLR. I am nevertheless recommending acceptance subject to minor revision through addressing the following points:
1. Notation: the paper is sloppy by using $D_{KL}$ as a shorthand for $E_q \log(q/p)$ even when $q$ and $p$ are not defined over the same space (see e.g. Eqs. 6 and 8 + Appendix). This can be very confusing to readers because this does not follow usual rules of KL divergence, such as non-negativity. In order for the paper to be acceptable, this would need to be clarified, ideally by reserving the notation $D_{KL}$ just for the KL divergence.
2. Related to the comment 11 of reviewer QSyx, the "True risk" term in Eq. (9) does not appear to be the true risk because it is an expectation over $q$, not $p$. This would need to be clarified.
3. As requested by the reviewers, please expand the discussion of the FEMNIST results and their significance.
4. Your response included a new figure (Fig. 1 in the response) that does not seem to be in the revised paper. Please consider adding it.
5. The reviewers raised several followup comments in the discussion after the author response. Please review and respond to these.

Typo: Appendix C refers to "Morkov's" inequality.
Decision: Accept (poster)