Personalized Federated Learning With Gaussian Processes
Authors: Idan Achituve
Aviv Shamsian
Aviv Navon
Gal Chechik
Ethan Fetaya
Venue: NeurIPS 2021 Poster (2021)
Keywords: Gaussian Processes
Federated Learning
Deep Kernel Learning

Abstract: Federated learning aims to learn a global model that performs well on client devices with limited cross-client communication. Personalized federated learning (PFL) further extends this setup to handle data heterogeneity between clients by learning personalized models. A key challenge in this setting is to learn effectively across clients even though each client has unique data that is often limited in size. Here we present pFedGP, a solution to PFL that is based on Gaussian processes (GPs) with deep kernel learning. GPs are highly expressive models that work well in the low data regime due to their Bayesian nature.
However, applying GPs to PFL raises multiple challenges. Mainly, GPs performance depends heavily on access to a good kernel function, and learning a kernel requires a large training set. Therefore, we propose learning a shared kernel function across all clients, parameterized by a neural network, with a personal GP classifier for each client. We further extend pFedGP to include inducing points using two novel methods, the first helps to improve generalization in the low data regime and the second reduces the computational cost. We derive a PAC-Bayes generalization bound on novel clients and empirically show that it gives non-vacuous guarantees. Extensive experiments on standard PFL benchmarks with CIFAR-10, CIFAR-100, and CINIC-10, and on a new setup of learning under input noise show that pFedGP achieves well-calibrated predictions while significantly outperforming baseline methods, reaching up to 21% in accuracy gain.

Personalized Federated Learning with Gaussian Processes
Idan Achituve
Aviv Shamsian Bar-Ilan University, Israel aviv. chamsian@biu..o ac.
Aviv Navon
Bar-Ilan University, Israel idan. achituve@biu.ac. ac
Bar-Ila -la University, Israel aviv navonebiu. ac
Gal Chechik Bar-Ilan University, Israel NVIDIA, Isreal
Ethan Fetaya Bar-Ilan University, Israel ethan. fethaan.utayybbiuaccc
gal. www.chhhhhhknnnnn ac
Abstract
Federated learning aims to learn a global model that performs well on client devices with limited cross-client communication. Personalized federated learning (PFL) further extends this setup to handle data heterogeneity between clients by learning personalized models. A key challenge in this setting is to learn effectively across clients even though each client has unique data that is often limited size. Here we present pFedGP, solution to PFL that is based on Gaussian processes (GPs) with deep kernel learning. GPs are highly expressive models that work well the low data regime due their Bayesian nature. However, applying GPs PFL raises multiple challenges. Mainly. GPs performance depends heavily on access good kernel function, and learning kernel requires large training set. Therefore, we propose learning a shared kernel function across all clients, parameterized by a neural network, with a personal GP classifier for each client. We further extend pFedGP to include inducing points using two novel methods, the first helps to improve generalization computational cost. We derive PAC-Bayes generalization bound on novel clients and empirically show that gives non-v non-vacuous guarantees. Extensive experiments standard PFL benchmarks with CIFAR-10, 10, CIFAR-100 and CINIC-10, and on a new setup of learning under input noise show that pFedGP achieves well-calibrated predictions while significantly outperforming bascline methods, reaching up 21% in accuracy gain.
the low data regime and the second reduces the
Introduction
In recent years, there is growing interest in applying learning in decentralized systems under the setup of federated learning (FL) [17 666. In FL, server node stores a global model and connects multiple end-devices ("clients"), which have private data that cannot be shared. The goal is to learn the global model in a communication-efficenn manner. However, learning a single shared model across all clients may perform poorly when the data distribution varies significantly across clients. Personalized Federated Learning (PFL) (67] addresses this challenge by jointly learning personalized model for each client. While significant progress had been made recent years, leading approaches still struggle in realistic scenarios. First, when the amount of data per client is limited,
even though this is one of the original motivations behind federated learning 4 51 72 Second,
when the input distribution shifts between clients, which is often the case, as clients use different devices and sensors. Last, when we require call-calibrated predictions, which is an important demand from medical and other cafety-critical applications.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Server
Client 1
Client S
VgL.
VeLc
0
Figure 1: pFedGP- learning shared deep kernel function with -picnt- GP models. Each client stores private data, possibly from a different distribution. The data is first mapped to an embedding space with a shared neural network across all clients. Then, using common kernels GP is applied the data of the client for model learning and inference. We illustrate the per-client kernel matrix (x,,, xj). Bold cells indicate a stronger covariance.
Here, we show how Gaussian Processes (GPs) with deep kernel learning (DKL) 80 is an effective alternative for handling these challenges. GPs have good predictive performance a wide range of dataset sizes 2 [1]. they robust to input noise 175. adapt shifts in the data distribution [48), and provide well-calibrated predictions 669]. While regression tasks are more natural for GPs, here we focus classification tasks for consistency with common benchmarks and learning procedures in the field; however, our approach is also applicable to regression tasks.
Consider naive approach that fits separate GP classifier to each client based on its personal data. Its performance heavily depends on the quality of the kernel, and standard kernels tend to work poorly in domains such images. A popular solution to this problem is to use deep kernel learning (DKL) [0). where a kernel applied to features outputted by a neural network (NN). Unfortunately, GPs with DKL can strongly overfit, often even worse than standard NNs [56). and thus negate the main benefit of using a GP. We solve this issue by jointly learning a shared kernel function across clients. As the kernel captures similarities between inputs, single kernel should work well across clients, while using a separate GP per client will give the required flexibility for personalization.
We adapt GP classifier recently proposed in 2 which uses the Pólya- 1aamm- augmentation 57 in a tree-strucee structure model the federated setting. We term our method pFedGP. We extend pFedGP by tailoring two inducing points (IPs) methods 70). The first helps generalization in the low data regime and, unlike common inducing point methods, does not reduce the computational costs. The second does focus on reducing the computational cost to make our approach scalable and work in low-resource clients. We also adjust previous PAC-Bayes generalization bounds for GPs (60 64 to include the Pólya-Gamma augmentation scheme. These bounds suitable for cases where the kernel is not learned, such as when new clients arrive after the shared NN was already Iearned.
Therefore, this paper makes the following contributions: (i) introduce pFedGP as a natural solution to PFL: (ii) develop two IP methods to enhance GP classifiers that use the Pólya-Gamma augmentation scheme and integrate them with pFedGP; (iii) derive a PAC-Bayes generalization bound on novel clients and show empirically that gives meaningful guarantees; (iv) achieve state-of-the thetfffth-attt arr results in wide array of experiments, improving accuracy by up to 21%
Related work
Federated learning. In FL, clients collaboratively solve learning task while preserving data privacy and maintaining communication efficiency [1] 34| 42 51
82 85]. FedAvg [51] is an early but effective FL approach that updates models locally and averages them into a global model. Several
54
optimization methods have been proposed for improving convergence in FL [1 45 77). Other approaches focus on preserving client privacy [3] 20 52 90), improving robustness to statistical
diversity (26 27 35 87 88), and reducing communication cost 61). These methods aim to leam global model across clients, limiting their ability deal with heterogeneous datasets.
'Our code is publicly available at wwt.ghwww......aaacaaattt////eee

---

Personalized federated learning. To overcome client heterogeneity, PFL aims to introduce some personalization for each client in the federation [39
73.. Recent methods include adapting multitask and model mixing, where clients Other approaches utilize different Personalization in FL has also been explored through clustering approaches in which similar clients within the federation have a greater effect one another [49 [66. Recently, [65] proposed learning a central hypernetworks that acts on client representation vectors for generating personalized models.
learning [18 67 meta-learning approaches 66
21
22
33
43
89
learn a mixture of the global and local models 4 17
27
regularization schemes to enforce soft parameter sharing 32
72
Bayesian FL. Some studies put forward Bayesian treatment to the FL setup. 8 used variational inference with Bayesian NNs. on the
proposed matching algorithm between local models based process to construct a global model. [14] extended Bayesian optimization to FL. setting via Thompson sampling. To scale the GP model they used random Fourier features. We use inducing points instead. (83 proposed federated learning framework that uses global GP model for regression tasks and without DKL.. Unlike this study, we focus on classification tasks with a personal GP classifier per client and advocate sharing information between clients through the kemnll. [74 used GPs in a client selection strategy. In (36) an approach based on stein variational gradient descent was suggested. This method does not scale beyond small-sized networks. 47 proposed multivariate Gaussian product mechanism to aggregate local models. As we will show, this method is less suited when the data heterogeneity between clients is large.
[76] 84
Gaussian process classification. Unlike regression, in classification approximations must be used since the likelihood is not Gaussian [59]. Classic approaches include the Laplace approximation 79. expectation- prrpppaationo [53] and least squares [62]. Recently, several methods were proposed based on the Pólya-Gamma augmentation [57] for modeling multinomial distributions [46], GP
classification [23 24 78), few-shot learning [69] and incremental learning (2]. Here we build on the
last approach. Classification with GPs is commonly done with variational inference techniques [30] here we wish to exploit the conjugacy of the model to take Gibbs samples from the posterior. This
approach yields well calibrated [69] and more accurate models 2
3 Gaussian processes background
We first provide brief introduction to the main components of model. Detailed explanations deferred to the Appendix. Scalars are denoted with lower-case letters (c.g.. I),, vectors with bold lower- case letters (c.g.. x), and matrices with bold capital letters (c.g.. X). In general, y = [y1,-. ..., yN]" is the vector of labels, and X E IR Nxd is the design matrix with N data points whose 7th row is x;.
Gaussian processes. GPs map input points to target output values via a random latent function f. I is assumed to follow Gaussian process prior S~ GP(m(x), k(x, x')), where the evaluation vector of ʃ on X, f = (x))....,, ('..,, (xx)) has a Gaussian distribution f ~ (i,, K) with means μ4 mmxx) and covariance K xx), The mean m(x) is often set to be the constant zero function, and the kernel (xx, x') is a positive semi-definite function. The target values are assumed to be independent when conditioned on f. For Gaussian process regression the likelihood is Gaussian, p(ylf) = [f, o²). Therefore, the posterior p(fly. X) is also Gaussian, and both the marginal and the predictive distributions have known analytic expressions. This is one of the main motivations behind using GPs, as most other Bayesian models have intractable inference.
Unfortunately, for Gaussian process classification (GPC) the likelihood, p(yl/), is not a Gaussian and the posterior does not admit a closed-fort -orm expression. One approach for applying GPs to binary classification tasks is the Pólya-Gamma augmentation (57] Using this approach, we can augment the GP model with random variables w from a P6lya-Gammn distribution, one for each example. As a result, p(f y, X. w) is a Gaussian density and p(w y, X. is Pólya-Gamma density. This allows to use Gibbs sampling to efficiently sample from the posterior pff, w w X) for inference and prediction. A key advantage of the
augmentation is that it benefits from fast mixing and has the ability of even a single value of w to capture much of the volume of the marginal distribution over function values [46). Full equations and further details the 76lya-Gamma augmentation scheme are given in A..
Deep kernel learning (DKL). The quality of the GP model heavily depends on the kernel function k(x,, x)). For many data modalities, such as images, common kernels are not a good measure of semantic similarity. Therefore, in [10 80) standard kernels are used over features outputted
3

---

by neural network go. For example, the RBF kernel xx,, exp
[[ge(x;) ( go (x))))
2
In regression, it is possible to directly backpropagate through the GP inference as it is given in closed-form. In our case, we use Fisher's identity [19] to obtain stochastic gradients [69].
Inducing points. GPs require storing and inverting kernel matrix on the entire training set which often limits its usage. A common solution this problem to use inducing point methods [0). The key idea is to replace the exact kernel with an approximation for fast computation. Usually, M < . pseudo-inputs are learned such that the main computational bottleneck is in inverting M x M matrices.
GP-Tree. We build on GP-Tree 2 recent GP classifier that shown to scale well with dataset size and the number of classes. GP-Tree turns the multi-class classification problem into a sequence of binary decisions along the tree nodes. Each node in the tree fits binary GP classifier based on the Pólya-Gamma augmentation scheme and the data associated with that node. The leaf nodes correspond to the classes in the dataset. The tree is constructed by first computing a prototype for each class and then recursively performing divisive hierarchical clustering on these prototypes to two clusters at each node. Further details are given in Appendix A.2
4 PFedGP: federated learning with Gaussian processes
Now we describe our approach for applying personalized federated learning (PFL) with Gaussian processes. First, extend GP-Tree to the FL setup and show how to use Gibbs sampling to learn the NN parameters. Then, we present two alternatives for this method that use inducing points. The first is for extremely limited-size datasets, while the second allows controlling the computational resources. We name our method pFedGP. An illustration of our method is given in Figure [
4.1 A full GP model
The training procedure follows the standard protocol in this field of server that holds the shared parameters (a NN). Let C denote the set of clients. For each client E C' we denote by De its local dataset of size N At each training iteration (round) the model sent to clients to perform kernel learning (|S) S [C]). Each client E updates its copy of the global model and then sends the updated model to the server. The server then averages over the updates to obtain a new global model.
51
We assume the existence
At each client we perform kernel learning in the following manner. We first compute the feature representation of the data samples associated with the client using the shared network. Then, we build the hierarchical classification tree as discussed in Section 3 Appendix A.2 In [2] the tree was built only once after a pre-training stage and the model parameters were learned using a variational inference approach. Here, we re-build the tree at each round using the most recent features and we use a Gibbs sampling procedure, as it allows this flexibility in building the tree and performs better when not prohibitive by computational limitations. Learning the network parameters 0 with the Gibbs sampling approach can be done with two common objectives, the marginal likelihood, and the predictive distribution.
We denote by X. the data associated with the tree node v, i.e., the data points which have on the path from the root node to their class leaf node. We denote by y. the binary label of these points, i... does their path go left or right at this node. And we denote by W. the Polya-Gamma random variables associated with node v. The marginal likelihood term for the full hierarchical classification tree is the sum of the separate marginal likelihood terms of all the nodes v in the tree:
Σ log / f
CLL D.)
Σ log (x.) =
No.y:/w....)pp(a (k)ppw,,)de
(I)
Similar to 669 we use gradient estimator based on Fisher's identity 19
MoLML LLCML(( De) Ef po polwolye. Volog (y X.) ,dw, 2
Σ
Σ v Volog po (y X.).
Here, ( ..., wi ! are samples from the posterior at node v. Due to the Gamma augmentation
po(yolwe (1) x.) is proportional to Gaussian density. The exact expression is give in Appendix A.2

---

To use the predictive distribution as an objective, in each training iteration, after building the tree model, at each node we randomly draw a portion from the (node) training data and use it to predict the class label for the remaining part. We denote with X,, and y. the training portion, x and y the input and the label of the point we are predicting, and pV" the path from the root node to the y" leaf node (i.e., the original class). Here we also take advantage of the independence between nodes to maximize the predictive distribution per node individually. The predictive distribution for a single data point:
C.P (0,
log pe ye
Σ log
/ polviile Yo, X., (x)ppw (y, Xow..
"EP''
v PM
(3)
We use an approximate-gradiee estimator based on posterior samples of we
VOL:P(0;'',,) ,1"
Σ
V elog (olyll
Y,, X.).
(4)
*EPY"
Where po Po (w! No, X.) x p )oo w x,, No, Xx)df* does not have an analytical
expression, but pe ( with known parameters. We then compute the predictive distribution by performing Gauss-Hermite integration over f*. See exact expression in Appendix A.2
( X,, x.) f (f [f, X, (o yv. X. is Gaussian
4.2 Augmenting the model with inducing points: sample efficiency
The GP model described in Section 4. works well in most situations. However, when the number of data points per client is small, performance naturally degrades. To increase information sharing between clients and improve the per-client performance, we suggest augmenting the model with global inducing points shared across clients. When sending the model from the server to a client, we also send the inducing inputs and their labels. To streamline optimization and reduce the communication burden, we define the inducing inputs in the feature space of the last embedding layer of the shared NN. Therefore, usually, their size will be negligible compared to the network size.
We denote by X the learned inducing inputs and by y their fixed class labels. They are set evenly across classes. During training, we regard only the set of inducing inputs-labels (X, y) as the available (training) data and use them for posterior inference. More formally, we first compute (f y, X) = (f X, X )pp [f(w,, y, X )df using its analytical expression for the actual training data and then compute the probability of y using Gauss-Hermite integration. Then we use Eq. B 4 for learning the network parameters and the inducing locations. At test time, to make full use of the training data, we combine the inducing inputs with the training data and use both to obtain the GP formulas and to make predictions. We note that with just using the inducing inputs at test time the model performs remarkably well, despite having almost no personalization component. See Appendix E.8 for further discussion.
One potential issue with using IPs in this manner is that it distorts the true class distribution. As a result, the classifier may be more likely to predict a www-pobbbbility class during test time. We address this issue by adjusting the output distribution. In general, let p(y, x) and g(y, x) be two distributions that differ only in the class probabilities, i.c. p(xly) = (xxly). the predictive distribution follows:
gly
)x) P(y
g(x (xx"ly")((" )a(v")
g(y")
(5) We use this to correct the GP predictions to the original class ratios at each node. We found in our experiments that this correction generally improves the classifier performance for class imbalanced data. As an example for this phenomena, consider a binary classification problem having 90 examples from the first class and 10 examples from the second class (therefore, q(3 0) = 0.9, and q0y 1) = 0.1). Assume we defined 50 inducing inputs per class, so now during test time the model sees 140 samples from the first class and 60 samples from the second class which corresponds
q(v0(x") x p(y*) P(y (x").
x*)
p(x* p(s"")
to probabilities p(y 0) 0.7 and ply = 1) = 0.3.
4.3 Augmenting the model with inducing points: computational efficiency
Learning the full GP model described in Section 4.1 requires inverting a matrix of size N₂ in the worst case (at the root node), which has ((N2) run-time complexity and O(N2) memory complexity.
5

---

Algorithm pFedGP. C' clients indexed by E number of local epochs; |S| number of sampled clients; M number of inducing inputs per class
Server executes:
Initialize shared network <-60 0o
Initialize M inducing inputs per class for all classes in the system # in pFedGP-IP variants only for each round 1, 2.. do:
Sample S clients uniformly at random
for each client
parallel:
1 Mf+1 + ClientUpdate((e..bl M,) # obtain updates from client e
Update 0c+1, Me+1 using FedAvg [51] update rule.
ClientUpdate((2, M): for each local epoch +- 1,.... E do:
e
Build GP tree classifier using the personal dataset D.
Update 0, .M using gradient-bar based optimization methods on D. with L or LPD return MM
Therefore, propose additional procedure based on inducing points to allow reduced complexity in low resource environments and scalability to larger dataset sizes.
This variant is based on the fully independent training conditional (FITC) method [70] The key idea to cast all the dependence on the inducing points and assume independence between the latent function values given the inducing points. Here for brevity, we omit the subscripts denoting the client and the tree node. However, all quantities and data points are those that belong to a specific client and tree node. Let X
R Ad xd denote the pseudo-inputs (defined in the embedding space of the last layer the NN), and f E R. the corresponding latent function values. Here as well, the inducing inputs are defined globally at the server level and they are set evenly across classes.
We assume the following GP prior p(f, f) N(00 KNN KNM]),
KMN KMM where MM the kernel between the inducing inputs, KNN is a diagonal matrix between the actual training data, KNM is the kernel between the data and the inducing inputs, and placed zero mean prior. The likelihood of the dataset when factoring the inducing variables and the Pólya-Gamma variables (one per training sample), and the posterior over f, both have known analytical expressions. We can then obtain the posterior and marginal distributions by marginalizing over f. Here we will present the posterior and marginal distributions:
p(f y.w, w X)
(ff N M - 1KMNA-'S 'S2-'K 'R, KNN KNA MM
B B-))KMN)..
Ply w X) N( k/0, A K KNM KNMK KMNN
Where = (w) Kj = y/ 1/2, A = Ω- + diag(KNN - KN MM KNN), and KMM + KMNNA A 'KNM. Importantly. we only need to invert M x M diagonal matrices. See full derivation in Appendix B During test time, we use f to get the posterior of f to compute the predictive distribution.
Now we can use either the marginal or the predictive distribution to learn the shared NN parameters and the inducing locations. The complexity of applying this procedure is reduced O(M2 + M) in run-time, and OMMN, + M²) in memory. While the (conditional) independence assumption between the latent function values may be restrictive, the full GP alternative in our experiments. Potentially, this can be attributed to the effect of sharing the inducing inputs among clients and the information that stores on f.
found this method to be comparable with
Generalization bound
It is reasonable to expect that after we learned the system new clients will arrive. In such cases, we would like to use PFedGP without re-training the kernel function. Under this scenario, we can derive generalization bounds concerning only the GP classifier without taking into account the fixed

---

neural network using PAC- Bayes bound [50]. Having meaningful guarantees can be very important in cafety-critical applications. The PAC-Bayes bound for GPC [64 (with the Gibbs risk):
Theorem 1. Given i..dd samples D {(x, Y of size Ne drawn from any data distribution over X x {-1,1}, GP posterior Q and a GP prior P, the following bound holds, where the probability is over random data samples:
PrD.(R(Q) > (Q) KL KL (Q). (Q). P,Q,)) 56.
(8)
Here, we have,
[(Q)= = (P
I" # f
(Q)
P fsign Si # va
No
<6, No, P.Q) [KT P|+ log 5
1 No
No + )) K KT (g, maz KTber
|pl
An important observation in 664 is that the KL-divergence between the posterior and prior Gaussian processes is equivalent to the KL-divergence between the posterior and prior distribution of their values on the Nº training samples. While [64] assumed Q to be Gaussian, this observation still holds even without this assumption. However, when Q is no longer Gaussian, is the case here, (T)(Q(f) || P(f)) no longer has closed-form expression. We can show that for the Pólya-Gamma augmentation:
((Q(() P()) (KL)( []]
log Q(w)
(10)
[ + Bo(r
Q ff)
where MI denotes the mutual information. Since ((
and P(f) are Gaussian, the K [T[((f) [ P(f) term has a close form expression so we only need to perform Monte- Carlo approximation on the expectation on w on the first element. the second expectation, Q(w) does not have known expression. To estimate it, given {ww,, ) sam- ples, we use (ww:) & N=I Ejz; Quwi (G)) Note that if the summation for j includes f.. it might result in biased estimator. Further details estimating K (Q)((()) || P(f))
0.6
0.5
Test Error
0.4
0.3
0.0
are Appendix C
0.0 0.1
0.5 0.6 Upper Bound
To assess the quality of the bound, we partitioned the CIFAR-10 dataset to 100 clients. We trained shared network using our full-GP variant on 90 clients and then recorded the generalization and test error on the remaining 10 clients four times, each with different training set size. Figure 2 shows the estimation the generalization error bound (o 0.01) the actual error the novel clients with the Gibbs classifier. First, we observe that indeed the bound is greater than the actual test error for all points and that it is not vacuous. There is strong correlation between the actual error and the bound. Secondly, unlike worst-case bounds (e.g.. VC-dimension),, this bound depends on the actual data and only the number of data points.
Figure 2: Test error vs an estimated up- per bound over 10 clients with varying degrees of a training set data size. Each dot represents combination of client and data size. In parenthesis the av- erage difference between the empirical and the test error.
Experiments
We evaluated pFedGP against baseline methods in various learning setups. We present the result for the following model variants: (i) pFedGP, the full GP model (Section[4.1l (ii) PFedGP-IP-date the model with IPs described in Section 4.2 and (iii) pFedGP-1P CCCPP-IP-cooputee the model with IPs described in Section 4.3 For pFedGP and pFeedGP-IP-coppue the results obtained by maximizing the predictive and marginal likelihood were similar, with slight advantage to the former. Therefore, we present here the results only for the predictive alternative and defer the results of the marginal alternative to the Appendix. Additional experiments, ablation study, and further analyses provided
7

---

Table I: Test accuracy (+ SEM) over 50, 100, 500 clients CIFAR-I0, CIFAR-100, and CINIC- 10.
The # samples/client indicates the average number of training samples per client.
CIFAR-10
CTFAR: 100
CINIC-10
clients
100
500
50
100
500
50
100
500
sampleskclient
800
400
80
800
400
80
1800
180
Local
86.2 +: 0.2
82.9
+
7.
52.1 +
309 4.3
0.2
56.9
0.7
46.4 0.1
FedAvg
56.4
59.7
0.5
23.6 0.5
0.2
24.0
204
45.6=04
44.7
45.7
FOLA
47
55.9
3.11
25.5 4.3
1.5
22.
18.7
4.3
38.3 0.3
FedPPe 4
83.8 1. 0.8
81.5 0.5
112
48.3
0.6
43.6
0.2
25.6
70.6
0.2
68.4
0.5
62.2 .05
LG-HiedAvg
0.77
43.6
59.5
pFedMe 72
$6.4 0.8
85.0
1.0.1.0.. 0.3
80.1
0.5
49.8
1 0.5
47.7
1 0.4
32.5
0.8
69.9
0.5
68.9
0.7
58.8
0.1
118
36.0
15.9
0.
59.3
PFedlIN 65
90.2
0.6
87.4.10.2 0.2
83.2
0.08
60.0
1
1.0
52.3
1
0.5
34.1
0.1
70.4
0.4
69.4
0.5
64.2
.05
Ours
PFedGP-IP-ddd
88.6 +: 0.2
8.
86.9 0.77
60.2
58.
55.7
69.8
0.2
68.3
0.6
67.6
DFooGR-IP-compute
89.9 + 0.6
88.8 + 0.1
8688 0.
6122
59.8
49.2
72.0
0.3
71.5 =
0.5
68.2
0.2
PFcdGP
88,8
87
633
0.1
61.3
50.6
71.8
0.3
in Appendix E
Unless stated otherwise, we report the average and the standard error of the mean (SEM) three random seeds of the federated accuracy, defined as the average accuracy across all clients and samples.
Datasets. All methods were evaluated on CIFAR-10, CIFAR-100 [38, and CINIC-I0 15 datasets.
CINIC-10 is more diverse since it combines images from CIFAR-IB 10 and ImageNet [66]
Compared methods. We compared our method against the following baselines: (1) Local, pFedGP full model on each client with private network and no collaboration with other clients; (2) FedAvg
[I standard FI. model with personalization component; (3) FOLA [47), Bayesian method that used a multivariate Gaussian product mechanism to aggregate local models; (4) FedPer 44, a
PFL approach that learns a personal classifier for each client on top of shared feature extractor; (5) LG-FedAvg [44), a PFL method that uses local feature extractor per client and global output layers; (6) pFedMe (72), a PFL method which adds Moreau-envelopes loss term; (7) FedU [8] recent multi-task learning approach for PFL that learns a model per client; (8) pFedHIN 665 a recent PFL approach that uses hypernetwork to generate client-specife networks.
Training protocol. We follow the training strategy proposed [65]. We limit the training process to 1000 communication rounds, in each we sample five clients uniformly at random for model updates. The training procedure different in the FOLA and pFedHN baselines, so we used an equivalent communication cost. In LG-FedAvvg we made extra 200 communication rounds after pre-training stage with the FedAvg model for 1000 communication rounds. In the local model, we performed 100 epochs of training for each client. In all experiments, we used LeNet-based network 40) having two convolution layers followed by two fully connected layers and an additional linear layer. We tuned the hyperparameters of all methods using pre-allocated held-out validation set. Full experimental details are given in Appendix D
6.1 Standard PFL setting
We first evaluated all methods in standard PFL setting (65 72
We varied the total number of clients in the system from 50 to 500 and we set the number of classes per client to two/ten/four for 1AA.-I///CIROOO//I is fixed, the number of samples per client changed accordingly. For each client, the same classes appeared in the training and test set.
respectively. Since the total number of samples the system
The results are presented in Table I They show that: (1) The performance of the local baseline is significantly impaired when the number of samples per client decreases, emphasizing the importance of federated learning in the presence of limited local data. (2) FedAvg and FOLA, which do not use personalized FL, perform poorly in this heterogeneous setup. (3) pFedGP outperforms or is on with previous state-of- saaeee-f-thheatt the-art approaches when local data is sufficient (e.g., 50 clients on all datasets). When the data per client becomes limited, pFedGP achieves significant improvements over competing methods; note the 9% and 21% difference in CIFAR-100 over 100 and 500 clients, respectively. (4) TredGP-IP-compur often achieves comparable results to pFedGP and often superior to pFedGP-IPP data. We believe that it can be attributed to the fact that in PFedGP-II-coomee computee

---

FiddAy
FOLA
FodiPr
L.G.FodAvg
pFalMe
(eCE: 186 0.300
10
1.0
L4
(cce: 3-304 ICC:
14
ROE: 2210
(9) U
(AI DUPeS
0.
02
0.4 06
01
1.0
5.
7.6
05
0.
0.
02
04
06
1.0
00
03
00
15
1.3
00
02
044
566
0.
3.0
FedU
PPedlIN
pFedGP-IP -P-ataa (ouns
DNedGP-IP-compunn (ours)
pFedGP (ours)
Ecb: E-21
DCE: S 222
DOR: O E23
0.287
60
06 9558
04425
044 Canfidence
Carfidence
Cont derce
Contidenie
Figure 3: Reliability diagrams on CIFAR-100 with 50 clients. Diagonal indicates perfect calibration. Each plot also shows the expected & maximum calibration error (ECE & MCE) and the Brier Score (BRI). Lower is better.
the training data take an active part in the GP inference formulas (Eq. 6). while in PFedGP-IP- the data impact in a weak manner only through the loss function. (5) PFedGP-IP-data is especially helpful when few samples per class are available, e.g.. CIFAR- 100 with 500 clients. That last point is further illustrated by decoupling the effect of the number of clients from that of the training set size. To illustrate that, in Appendix E.2 we fixed the number of clients and varied the number of training samples per class. From this experiment, we deduced that both factors (individually) contribute to pFedGP success.
A desired property from PFL classifiers is the ability to provide uncertainty estimation. For example, in decision support systems, such as in healthcare applications, the decision-maker should have an accurate estimation of the classi- fier confidence in the prediction. Here, we quan- tify the uncertainty through calibration. Figure 3 compares all methods both visually and using common metrics 7 25
Table 2: Test accuracy (# SEM) over 100 clients on noisy CIFAR-100. We also provide the relative accuracy decrease (%) w.r.t. the performance on the original CIFAR- 100 data (see Table T.
Method
Accuracy
Decrease (%)
FedPer
28.1 + 0.9
-35.6
LG-FedAvg
26.9 1 0.9
-28.3
[55 on the CIFAR- 100 dataset with clients. Expected calibration error (ECE) measures the weighted average be- tween the classifier confidence and accuracy. Maximum calibration error (MCE) takes the maximum instead of the average. And, Brier score (BRI) 7 measures the average squared error between the labels and the prediction prob- abilities. The figure shows that pFedGP classifiers best calibrated across all metrics in almost all cases. We note that with temperature scaling, the calibration of the baseline methods can be improved 25): however, choosing the right temperature requires optimization over separate validation set, which our model does not need. Additional calibration results, including temperature scaling, are presented in Appendix R.9
pFedme 72
33.2 + 0.6
-30.4
FedU [8]
35.0 + 0.2
-2.8
pFedHN 65
38.9 + 0.5
-25.7
Ours
PFedGP-IP-data TredGP-IP-compute
45.0 0.3
-23.1
47.1 .05
-21.2
pFedGP
49.5 0.1
-19.2
6.2 PFL with input noise
In real-world federated systems, the clients may employ different measurement devices for data collection (cameras, sensors, etc.), resulting in different input noise characteristics per client. Here, we investigate pFedGP performance in this type of personalization. To simulate that, we partitioned CIFAR- 10/100 to 100 clients similar to the protocol described Section 6.1 we defined 57 unique distributions of image corruption noise [29]. and we assigned noise model to each client. Then for each example in each client, we sampled a corruption noise according to the noise model allocated to that client. Here show the results for the noisy CIFAR-100 100 dataset in Table 2 Further details

---

on the perturbations performed and result for the noisy CIFAR-10 10 are given in the Appendix We observe a significant gap favor of the pFedGP variants compared to baseline methods. Note that using global inducing points is slightly less beneficial in this case since they are defined globally and therefore are not tied a specific noise type as the real client data
6.3 Generalization to -uttof-distrbbution (OOD) novel clients
FL are dynamic systems. For example, novel clients may enter the system after the model was trained, possibly with a data distribution shift. Adapting to a new OOD client is both challeng- ing and important for real-world FL systems. To evaluate PFedGP in this scenario, we followed
pFedGP-
pFedGP -P computt
Generalization Gap
pFedGP
-5
DfedHN
-10-
pfediee
FedPer
-15
the learning protocol proposed in [65] We par-
-20-
titioned the CIFAR-10 dataset into two groups.
The data in the first group was distributed be-
-30
tween 90 clients for model training. The remain-
0.4 0.8
ing data from the second group was distributed between an additional 10 clients that were cluded during training. Within each group, we set the class probabilities in each client by sam- pling from a Dirichlet distribution with the same parameter, For the training group, we set a = 0.1, trained the shared model using these clients, and froze it. Then, evaluated the models on the second group by varying a E {.1, 25, 5,, .75, 1}. on the remaining 10 clients. As a moves away from 0.1 the distribution shift between the two groups increases, resulting more challenging OOD clients. Figure 4 reports the generalization gap as a function of the Dirichlet parameter a. The generalization gap computed by taking the difference between the average test accuracy of the (ten) novel clients and the average test accuracy of the (ninety) clients used for training. From the figure, here as well, pFedGP achieves the best generalization performance for all values of a. Moreover, unlike baseline methods, pFedGP does not require any parameter tuning. Several baselines were excluded from the figure since they had a large generalization gap.
Figure 4: Generalization to novel clients on CIFAR-10.
7 Conclusion
In this study, we proposed pFedGP, novel method for PFL. pFedGP learns kernel function, parameterized by a NN, that is shared between all clients using a personal GP classifier on each client. We proposed three variants for pFedGP, full model approach that generally shows the best performance and two extensions to it. The first is most beneficial when the number of examples per class small while the second allows controlling the computational requirements of the model. We also derived PAC-Bayes generalization bound on novel clients and empirically showed that it gives non-vacuous guarantees. PFedGP provides well-calibrated predictions, generalizes well to OOD novel clients, and consistently outperforms competing methods.
Broader impact: Our method shares the standard communication procedure of FL approaches, where no private data is directly communicated across the different nodes in the system. This protocol does not explicitly guarantee that private information pFedGP is particularly useful for clients with little data, and for clients that have strongly different distribution. This has great potential to improve client personalization in real-world systems, and do better at handling less common data. The latter is of great interest for decision support systems in sensitive domains such as health care legal.
be inferred at this time. As show,
Acknowledgements
This study was funded by a grant to GC from the Israel Science Foundation (ISF 737/2018), and by an equipment grant to GC and Bar-Ilan University from the Isracl Science Foundation (ISF 2332/18). IA was funded by a grant from the Israeli innovation authority, through the AVATAR consortium.
'The noisy CIFAR- 10/100 datasets available http://fanncchuuuugggghbbbboeoep
10

---

References
[1] Sawsan Abdulrahman, Hanine Tout, Hakima Ould-Slimane, Azzam Mourad, Chamseddine Talhi, and Mohsen Guizani. A survey federated learning: The journey from centralized to distributed on-site learning and beyond. IEEE Internet of Things Journal, 8(7)55476-5477, 2021.
[2] Idan Achituve, Aviv Navon, Yochai Yemini, Gal Chechik, and Ethan Fetaya. GP-Tree: A Gaussian process classifier for few-shot incremental learning. In Proceedings of the 38th International Conference on Machine Learning, pages 54-65. PMLR, 2021.
[3] Naman Agarwal, Ananda Theertha Suresh, Felix Xinnan X Yu, Sanjiv Kumar, and Brendan McMahan. cpSGD: Communication--ffiiieenn and cuffereetiallly-riivae distributed SGD. In Advances in Neural Information Processing Systems, pages 7564-7575, 2018.
[4] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint
2019.
[5] David Arthur and Sergei Vassilvitskii. k-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027-1035, 2007.
[6] Harkirat Singh Behl, Atiim Güneş Baydin, and Philip HS Torr. Alpha MAML: Adaptive model-agnostic meta-leaming. arXiv preprint arXiv:/905007453 2019.
[7] Glenn W Brier. Verification of forecasts expressed in terms of probability, Monthly weather review, 78(1):1-3, 1950.
[8] Thang D Bui, Cuong Nguyen, Siddharth Swaroop, and Richard E Turner. Partitioned variational inference: A unified framework encompassing federated and continual learning. arXiv preprint 1rXiv:181.11122 2018
[9] Dongqi Cai, Qipeng Wang, Yuanqiang Liu, Yunxin Liu, Shangguang Wang, and Mengwei Xu. Towards ubiquitous learning: first measurement of on-device training performance. In Proceedings of the 5th International Workshop on Embedded and Mobile Deep Learning, pages
31-36, 2021.
[10] Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Mare Peter Deisenroth. Manifold Gaussian processes for regression. 2016 International Joint Conference on Neural Networks (IJCNN), pages 3338-3345. IEEE, 2016.
[11] Shuxiao Chen, Qinqing Zheng, Qi Long, and Weijie J Su. A theorem of the alternative for personalized federated learning. arXiv preprint arXiv:2103.0/900) 2021.
[12] Luca Corinzia, Ami Beuret, and Joachim M Buhmann. Variational federated multi-task learning. arXiv preprint arXiv: 966606668, 2019.
[13) Xinyan Dai, Xiao Yan, Kaiwen Zhou, Han Yang, Kelvin KW Ng, James Cheng, and Yu Fan. Hyper-sphere quantization: communication-effiice SGD for federated learning. arXiv preprint 1911v:/91100463 2019.
[14] Zhongxiang Dai, Bryan Kian Hsiang Low, and Patrick Jaillet. Federated Bayesian optimization via Thompson sampling. Advances in Neural Information Processing Systems, 33, 2020.
[15] Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos J Storkey. CINIC-10 is not Imagenet or CIFAR-10. arXiv preprint arXiv: 180000355. 2018.
[16] Jia Deng, Wei Dong. Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-28 255. lece, 2009.
[17] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, arXiv preprint 13444 2020.

---

[18] Canh T Dinh, Tung T Vu, Nguyen H Tran, Minh N Dao, and Hongyu Zhang. FedU: A unified framework for federated multi-task learning with Laplacian regularization. arXiv preprint arXiv:2102.07144 148, 2021.
[19] Randal Douc, Eric Moulines, and David Stoffer. Nonlinear time series: Theory, methods and applications with examples. CRC press, 2014.
[20] John Duchi, Michael Jordan, and Martin Wainwright. Privacy aware learning. Journal of the ACM (JACM), 61(6):1-57, 2014.
[21] Alireza Fallah, Aryan Mokhtari, and A. Ozdaglar. Personalized federated learning: A meta- learning approach. Advances in Neural Information Processing Systems, 2020.
[22] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model-agnostic meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1082-1092. PMLR, 2020.
[23] Théo Galy-Fajou, Florian Wenzel, Christian Donner, and Manfred Opper. Multi-class Gaussian process classification made conjugate: Efficient inference via data augmentation. In Uncertainty in Artificial Intelligence, pages 755-765. PMLR, 2020.
[24] Théo Galy-Fajou, Florian Wenzel, and Manfred Opper. Automated augmented conjugate inference for non-conjugate gaussian process models. In International Conference on Artificial Intelligence and Statistics, pages 3025-3033 PMLR, 2020.
[25] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[26] Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint
14425, 2019.
[27] Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.055116 2022..
[28] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[29] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[30] James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable variational Gaussian process classification. In Artificial Intelligence and Statistics, pages 351-360. PMLR, 2015.
[31] T. H. Hsu, Hang Qi, and M. Brown. Measuring the effects of non-identical data distribution for federated visual classification. ArXiv, 9tts//999,0635,, 2019.
[32] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang. Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-llI data. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[33] Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning, arXiv preprint arXiv: /909, 12488, 2019.
[34] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv: 1xXiv:1912.04977 2019.
[35] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.
[36] Rahif Kassab and Osvaldo Simeone. Federated generalized Bayesian learning via distributed Stein variational gradient descent. arXiv preprint 2000002009.06-11, 2020.
12

---

[37] Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimiza- tion: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527. 2016.
[38] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[39] V. Kulkarni, Milind Kulkarni, and A. Pant. Survey of personalization techniques for federated learning. 2020 Fourth World Conference Smart Trends in Systems, Security and Sustainability (WorldS4), pages 794-797, 2020.
[40] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, 86(1 ((:2)1):277223311 2324, 1998.
[41] Tian Li, Anit Kumar Sahu, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, and Virginia Smith. On the convergence of federated optimization in heterogeneous networks. arXiv preprint /rXiv:/812.06122 2018.
[42] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020.
[43] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to learn quickly for few-shot learning. arXiv preprint urXiv:1701 707.09835, 2017.
[44] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis -hhil-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint 2rXiv:2001.01523, 2020.
[45] Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local SGD. In International Conference on Learning Representations, 2019.
[46] Scott W Linderman, Matthew Johnson, and Ryan P Adams. Dependent multinomial models made easy: stick breaking with the Pólya-Gamma augmentation. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 3456-3464, 2015.
[47] Liangxi Liu and Feng Zheng. A Bayesian federated learning framework with multivariate gaussian product. arXiv preprint arXiv:2102.01936. 2021.
[48] Wesley Maddox, Shuai Tang, Pablo Moreno, Andrew Gordon Wilson, and Andreas Damianou. Fast adaptation with linearized neural networks. In International Conference on Artificial Intelligence and Statistics, pages 2745.2745. PMLR, 2021.
[49] Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh. Three approaches for personalization with applications to federated learning. ArXiv, abs/2002.10619, 2020.
[50] David A McAllester. PAC-Bayesian stochastic model selection. Machine Learning, 51(1):5-21, 2003.
[51] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication--ffiiinn learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
[52] H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2018.
[53] Thomas Peter Minka. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, 2001.
[54] Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. survey on security and privacy of federated learning. Future Generation Computer Systems, 1 115:619-666 640, 2021.
13

---

[55] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[56] Sebastian W. Ober, Carl E. Rasmussen, and Mark van der Wilk. The promises and pitfalls of deep kernel learning. CoRR, 2021.
[57] Nicholas G. Polson, James G. Scott, and Jesse Windle. Bayesian inference for logistic models using Pólya-Gamma latent variables. Journal of the American Statistical Association, pages 1339-1349, 2013.
[58] Joaquin Catnonerro-Cnndlll and Carl Edward Rasmussen. A unifying view of sparse approxi- mate Gaussian process regression. The Journal of Machine Learning Research, 6:1999-1959 2005.
[59] Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian Processes for Machine Learning. The MIT Press, 2006.
[60] David Reeb, Andreas Doerr, Sebastian Gerwinn, and Barbara Rakitsch. Learning Gaussian processes by minimizing PAC-Bayesian generalization bounds. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 3341-3351, 2018.
[61] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. FedPAQ: A communication-effan effficient federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics, pages 2021- 2031. PMLR, 2020.
[62] Ryan Rifkin and Aldebaro Klautau. In defense of one-vs-all classification. The Journal of Machine Learning Research, 5:101 2004.
[63] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[64] Matthias Seeger. PAC-Bayesian generalisation error bounds for Gaussian process classification. Journal of machine learning research, 3(0ct):233-269, 2002.
[65] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning. PMLR, 2021.
[66] Reza Shokri and Vitaly Shmatikov.
deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1310-1321, 2015.
[67] Virginia Smith, Chao-Kai Chiang. Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4427-4437, 2017.
[68] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4080-4090, 2017.
[69] Jake Snell and Richard Zemel. Bayesian few-shot classification with one-vs-each Polya-Gamma augmented Gaussian processes. In International Conference on Learning Representations, 2021.
[70] Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using useudo-inputs In Advances in Neural Information Processing Systems, pages 1257-1264 MIT Press, 2006.
[71] Sebastian U Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2018
[72] Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized federated learning with Moreau envelopes. Advances in Neural Information Processing Systems, 33, 2020.
14

---

[73] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. arXiv preprint arXiv:2103.00711 2021.
[74] Minxue Tang, Xuefei Ning, Yitu Wang, Yu Wang, and Yiran Chen. Fedgp: Correlation-based active client selection for heterogeneous federated learning. arXiv preprint arXiv:2103. 13822, 2021.
[75] Carlos Villacampu-Calve, Bryan Zaldívar, Eduardo C Lobato. Multi-class Gaussian process classification with noisy inputs. Journal of Machine Learning Research, 22(66)11-52 2021.
and Daniel Hernández-
[76] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitriss Papailiopoulos, and Yasaman Khaz- aeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2019.
[77] Jianyu Wang and Gauri Joshi. Cooperative SGD: A unified framework for the design and analysis of communication-effin -ffficient SGD algorithms. In ICML Workshop on Coding Theory for Machine Learning. 2019.
[78] Florian Wenzel, Théo Galy Majo-, Christian Donner, Marius Kloft, and Manfred Opper. Efficient Gaussian process classification using Pôlya-Gamma data augmentation. In The AAAI Conference on Artificial Intelligence, pages 5417-5424. AAAI Press, 2019.
[79] Christopher KI Williams and David Barber. Bayesian classification with Gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20() 1342232332211 1342-1351, 1998.
[80] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2016.
[81] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2594 26022, 2016.
[82] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
[83] Feng Yin, Zhidi Lin, Qinglei Kong, Yue Xu, Deshi Li, Sergios Theodoridis, and Shuguang Robert Cui. FedLoc: Federated learning framework for data-driven cooperative localization and location data processing. IEEE Open Journal of Signal Processing, 1:187-215, 2020.
[84] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pages 7252-7261 PMLR, 2019.
[85] Chen Zhang, Yu Xie, Hang Bai. Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge- Systems, 116:15 2021.
[86] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. arXiv preprint arXiv:2012.08565, 2020.
[87] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-lID data. arXiv preprint arXiv:1806.000 00582, 2018
[88] Fan Zhou and Guojing Cong. On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimization. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 3219-3227, 2018
[89] Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, and Jiashi Feng. Efficient meta learning via minibatch proximal update. Advances in Neural Information Processing Systems, 32:1534-1544, 2019.
[90] Wennan Zhu, Peter Kairouz, Brendan McMahan, Haicheng Sun, and Wei Li. Federated heavy hitters discovery with differential privacy. International Conference on Artificial Intelligence and Statistics, pages 3837-3847. PMLR, 2020.
15


Reviews:
[{"review_id": "yGOMm13jg2f", "reviewer": "Reviewer_NyvV", "summary": "This work considers the task of learning user personalised predictive models in the federated learning setting. It is realised via a local Gaussian Process (GP) on each client which operates on top of features extracted from a neural network that is shared across the entire federation, named pFedGP. The authors motivate the usage of GPs via their Bayesian nature which allows for good performance in the low data regime (common in federated learning) as well as the extraction of prediction probabilities that are calibrated (crucial for safety-critical applications of FL). \n\nSince the authors operate on federated classification tasks,  they employ the GP classification model proposed at [1], which decomposes a multi-class prediction problem into a series of binary predictions. The benefit of this is that one can use the Polya-Gamma augmentation scheme, which can allow for efficient sampling of the posterior distributions, necessary for learning the hyper parameters of the model. The authors then  introduce two additional variants of their overall model which employ inducing points, a common technique in the GP literature. The first one is motivated for improving the performance of the model in the case of limited data, whereas the second one is about making the model more scalable in the case of large local datasets. Finally, the authors also provide a PAC-Bayesian generalization bound for the model performance one new clients that enter the federation. \n\nThe paper concludes with a series of experiments that compare the performance of pFedGP against several baselines for federated learning and personalised federated learning. \n\n[1] GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning, Achituve et al., 2021", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work tackles the problem of personalisation in federated learning which is an important and active area of research in federated learning. The parts of the proposed approach are not novel themselves, however their combination for the task of personalised FL is new. The discussion of the method is relatively clear, and it is nice to see that the uncertainties provided by pFedGP are well calibrated. \n\nDespite these positive aspects, I cannot recommend for accepting the paper as is currently and below I elaborate why:\n- The overall method seems quite computationally intensive, as one needs to perform clustering and several iterations of Gibbs sampling in order to learn the model. Since the compute is done locally in resource constrained devices in federated learning, I believe it is important for the authors to contrast the computational requirements (e.g., in terms of floating point operations) of their method against the more traditional NN training of, e.g., FedAvg. Furthermore, it would also be helpful if the authors include a pseudo-code of the overall federated training procedure with the different variants of pFedGP. \n- The generalisation bound provided in the submission is independent of a core part of the model, the neural net. As a result, one can wonder about its usefulness in practice where a wide variety of neural networks might be employed. For the current experiments it seems to work reasonably well, however the network used is a simple LeNet and the datasets (CIFAR10/100, CINIC-10) are quite  for the complex for this architecture (so it is harder to overfit). In an ideal case, it would be good if the authors include the neural network in the generalisation bound. As this might be harder to do, it would perhaps be worthwhile if the authors can show the behaviour of the bound with more complicated networks that have more parameters. Furthermore, there are some approximations involved in computing the bound with the Polya-Gamma augmentation (i.e., the expectation over $Q(w)$ and estimation of the mutual information). It is a bit unclear how finite-sample estimates of these quantities affect the actual bound and, ideally, they should be taken into account when providing the actual guarantee. \n- The experimental section and setup is a bit unclear with respect to a couple of important FL training hyperparameters. The authors use non-iid settings for the experiments. In prior work [2], it has been shown that in such scenarios, doing less epochs of local optimization (denoted as $E$) and using more clients per round can be beneficial for the final performance of the model when using FedAvg (which makes sense, as more local epochs can push each of the local models further away from each other, as each client has its own specialised dataset). Going over the authors’ appendix, it seems that for the baselines they use $E \\approx 20$ but $E \\in set(1, 3)$ for pFedGP. In fact, the reference [2] has managed to train a Lenet type of network with FedAvg on a similar non-iid version of CIFAR 10  and got up to ~73% performance, which is much better than the ~60% this work reports. It would be good if the authors adapt the hyperparameters appropriately so that the comparison is more meaningful. Finally, for a better personalised model performance baseline that is quite simple you should consider the performance of the locally fine-tuned model when training with FedAvg, see [3].\n- One result that was relatively surprising was that FedPer had much worse performance than pFedGP, as they are quite similar in a sense; they both use the same backbone, except that one uses a linear classifier whereas the other uses a GP. It would be great if the authors can explain why this is the case. Furthermore, upon inspecting the author’s code it seems that they use two convolutional followed by two fully connected layers + one extra embedding layer, before feeding the embeddings to the GP. This is a larger model than the traditional lenet which, after the two convolutional layers, uses a single fully connected  layer and then the linear classifier. What kind of backbone did the authors use for their baselines? Did it have the same number of layers? (Unfortunately, I couldn’t find the code for the baselines in the supplementary). I would suggest that, for future comparisons, the authors, use a comparable architecture to lenet, in that they use the output of the first fully connected layer as an embedding for the GP.\n\nFinally, below I have some further comments about the clarity of this work and how it could be improved:\n- At line 192 in the main text the authors mention that inducing points can distort the true class distribution for pFedGP and that they use reweighting to combat this effect. In order to improve the clarity of the submission, I believe that the authors should provide an example of this event and why it happens. Furthermore, it would be beneficial if they explain how critical is this modification and how it is done in practice (i.e., what is $q(y, x)$, $p(y, x)$?).\n- On lines 301-302 the authors mention that pFedGP-data is especially helpful when few datapoints per class are available, nevertheless, this seems to be the case only for CIFAR100 but not for CIFAR10 / CINIC-10. It would be helpful in the authors explain why. Finally, it seems that the inducing points were fixed to 100 for all the experiments? How critical of a hyper parameter is the number of inducing points for the performance of pFedGP?\n\n\n[2] Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification, Hsu et al., 2019\n\n[3] Improving Federated Learning Personalisation via Model Agnostic Meta Learning, Jiang et al., 2019", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}, {"review_id": "khxI7xlq6tt", "reviewer": "Reviewer_ys9C", "summary": "This paper proposes to use Gaussian processes for personalized federated learning (PFL), and use deep kernel learning to allow all clients to jointly learn a shared kernel. Two inducing points-based extensions are also introduced to facilitate more information sharing and reduce the computational cost respectively.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an interesting application of GP with deep kernel learning to federated learning, and has shown competitive empirical performances for the proposed algorithm. My main concern about the current paper is regarding the technical novelty, in particular, most of the technical contributions seem to adapted results from existing GP methods, and the proposed methods seem like pure applications of GP to FL.\n\nStrengths:\n- In the experiments, a comprehensive set of existing methods have been compared with. The experimental results in Table 1 are very compelling, and the results in Figure 3 also look very nice, clearly demonstrating the advantage of GP in terms of uncertainty calibration.\n\nWeaknesses/Questions:\n- I think the use of GP for PFL is not well motivated. I think the essence of Personalized FL is to incorporate some amount of personalized information (learned from a client's local data) into a jointly learned ML model. In this sense, GP is only one of many possible choices. In fact, the most natural choice in my opinion is to use a mixture of global and local models, which has also been mentioned in related works. So I think a better and clearer motivation is needed.\n- My overall impression is that the technical contents of this paper is almost exclusively about GP, rather than FL. For example, in Section 4.1, the only place about FL is the paragraph starting from line 149 where we can use standard FL to learn a neural network which is used to extract representation for kernel learning; in fact, all equations in this section are from existing GP techniques if I understood correctly. Sections 4.2 and 4.3 simply treat the inducing inputs and outputs as additional parameters to be shared among clients (again the equations such as equation 6 and 7 in these two sections are from existing results on GP), and the generalization bound in Section 5 is a pure GP result. Therefore, in my opinion, this paper is an application paper which applies existing results on GP (with some modifications) to FL, which hence may limit the technical contributions.\n- Section 4.2: how do you learn the inducing points? The method in this section uses inducing points as the parameters for information sharing among different clients to overcome the nonparametric nature of GPs, so the paper [1] listed below should be cited here which uses another common technique (random Fourier features) for the same purpose.\n- Section 4.3: lines 208-209, the inducing points are also defined globally by the server, so it's unclear to me what's the difference between the methods in Sections 4.2 and 4.3?\n- Section 5: As mentioned above, the results in this section seem to have nothing to do with FL. Therefore, I doubt whether it will add much value to the current work as a paper for PFL.\n- Section 5: just to clarify, is it correct that Theorem 1 is an existing result from [56], and your contribution here is to calculate the upper bound for your specific choice of GP classifier with Polya-Gamma augmentation? Lines 248-249: why have you chosen this approximation for $Q(\\boldsymbol{\\omega}_i)$? Is it like a marginalization over the $\\boldsymbol{f}_i$'s?\n- Experiments, is there a way to quantify and compare your computational cost with other methods? Because GPs are usually associated with large computational cost compared with parametric models like neural networks. One of the baselines is \"Local\" with uses a GP together with a locally learned kernel network, so have you tested the performance of standard GP (without kernel learning)? Not sure if I understood correctly, in line 292, it is mentioned that for CIFAR-10, every agent only has access to data belonging to two classes; however, in Table 1, how can the Local method still achieve an accuracy of above 70%? The same applies to the other two datasets.\n- The paragraph starting from line 127: I feel that more technical background on GP-Tree is needed here since it's the backbone for all algorithms in this work.\n\n[1] Federated Bayesian optimization via Thompson sampling. Dai, et. al. NeurIPS 2020.\n\nSome minor comments:\n- line 119: one of the $\\mathbf{x}_i$'s inside exp should be $\\mathbf{x}_j$\n- line 269: \"federate\" -> \"federated\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "K3LYVZlh24E", "reviewer": "Reviewer_anUe", "summary": "This paper presents a new framework for personalized federated learning via Gaussian process classification (GPC) model. The key idea is to have each client node maintain its own GPC model but share the NN feature map f(x). Here, the GPC model of client c is equipped with kernel k_c(x, x') = k_c( f(x), f(x'); w_c)  with personalized parameters w_c. \n\nThus, per iteration, a subset of clients is selected. Each client c downloads the shared f(x) from server and re-build its GPC model (i.e., re-fitting for w_c) using local data D_c. A local updated feature map f(x) (via gradient propagation) is then computed for each client and communicated to the server, which averages the updated model to form a new feature map f(x).\n\nThe proposed framework is evaluated on 3 image datasets: CIFAR-10, CIFAR-100 and CINIC-10. The reported performance shows that it outperforms a selected set of baselines substantially when the amount of data per client is small & heterogeneous.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "NOVELTY & SIGNIFICANCE\n\nI believe this is the first personalized federated learning framework that emphasizes on separating between learning the model (which is private) and learning feature representation (which is shared and can be learned via FL). This is an inherent separation the construction of a GP model which makes it a good model base. In addition, choosing GP as a base model is also a practical choice seeing that GP is often more effective on low-data regime (thus, being more suitable to provide personalized performance to client with little data). Thus, concept-wise, I consider the proposed method novel. \n\nOn the other hand, the proposed method appears to combine straight-forwardly a bunch of existing techniques (i.e., a vanilla local GP training method, direct adoption of FedAvg to learn a shared feature map & a vanilla PAC bound to assess the generalized performance on novel clients) so algorithmic-wise, there is little novelty & significance. That being said, I also believe the significance of the proposed method is more on the practical side, regarding its extensive empirical studies, on which I will further comment below.\n\nQUALITY\n\nThe paper is technically correct. I have gone through its technical development & derivation and have not found any flaws. I do, however, find the contributions of the paper somewhat orthogonal at certain parts: \n\nFirst, it is not clear to me the role of a GP-tree model in this context -- I thought any GPC would do so if we are to pick a more sophisticated model, there must be a reason. I guess it has to do with the performance but I am not sure why so could the author elaborate more on this? \n\nLikewise, I also do not see the advantage of using Polya-Gamma augmentation over normal approximation techniques for GPC. Please also discuss this in the rebuttal.\n\n---\n\nRegarding the experiments, I am a bit confused that pFedGP-IP-data is inferior to pFedGP-IP-compute. This is somewhat strange since pFedGP-IP-compute only uses the shared inducing points so there is not much to personalize. In contrast, pFedGP-IP-data includes both inducing points and local data points for inference so it is not clear to me why pFedGP-IP-data is inferior to pFedGP-IP-compute. Could the authors elaborate more on this too?\n\nIn addition, I find the personalized FL work in [16] very relevant & is directly applicable to the setting here. Could the author provide extra, minimal comparison with [16] in the rebuttal?\n\n---\n\nRegarding the theory, I suppose the point of section 5 is that if we use the feature map directly to build a personalized GPC to a novel client, we can make use of the PAC-bound for GPC in [56] to (empirically) measure the generalization gap. While this is not exactly a novel point, I appreciate the empirical analysis that makes use of this bound to showcase the enhanced OOD generalizability of the proposed model on novel clients, especially there is an increasing distributional shift. This speaks well to the practical significance of the proposed method.\n\nCLARITY\n\nThe paper is very well-written. However, I feel that to a more general audience, the lack of background cover on Polya-gamma & GP-tree might impair readability. I'd suggest cutting down on section 5 to make room for those background detail. \n\nREVIEW SUMMARY\n\nIn general, I enjoy reading this paper. The proposed method provides quite a refreshing perspective and it has also been evaluated meticulously. That being said, I do have some questions for the authors & some of which is rather important (e.g., comparison with [16]) and I hope the authors would address those in the rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have included a broader impact statement elaborating on the societal impact of the proposed work. I find that acceptable & have no further concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "No concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper adopts and adapts a number of known GP learning techniques to develop a method for personalised federated learning. The proposed method is computationally intensive, but performs well in practice and even comes with theoretical utility analysis. After author feedback, all reviewers appear happy with the paper and recommend acceptance, assuming the authors revise the paper to incorporate the material from the author feedback.
Decision: Accept (Poster)