Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to use Gaussian processes for personalized federated learning (PFL), and use deep kernel learning to allow all clients to jointly learn a shared kernel. Two inducing points-based extensions are also introduced to facilitate more information sharing and reduce the computational cost respectively.
main_review: The paper proposes an interesting application of GP with deep kernel learning to federated learning, and has shown competitive empirical performances for the proposed algorithm. My main concern about the current paper is regarding the technical novelty, in particular, most of the technical contributions seem to adapted results from existing GP methods, and the proposed methods seem like pure applications of GP to FL.

Strengths:
- In the experiments, a comprehensive set of existing methods have been compared with. The experimental results in Table 1 are very compelling, and the results in Figure 3 also look very nice, clearly demonstrating the advantage of GP in terms of uncertainty calibration.

Weaknesses/Questions:
- I think the use of GP for PFL is not well motivated. I think the essence of Personalized FL is to incorporate some amount of personalized information (learned from a client's local data) into a jointly learned ML model. In this sense, GP is only one of many possible choices. In fact, the most natural choice in my opinion is to use a mixture of global and local models, which has also been mentioned in related works. So I think a better and clearer motivation is needed.
- My overall impression is that the technical contents of this paper is almost exclusively about GP, rather than FL. For example, in Section 4.1, the only place about FL is the paragraph starting from line 149 where we can use standard FL to learn a neural network which is used to extract representation for kernel learning; in fact, all equations in this section are from existing GP techniques if I understood correctly. Sections 4.2 and 4.3 simply treat the inducing inputs and outputs as additional parameters to be shared among clients (again the equations such as equation 6 and 7 in these two sections are from existing results on GP), and the generalization bound in Section 5 is a pure GP result. Therefore, in my opinion, this paper is an application paper which applies existing results on GP (with some modifications) to FL, which hence may limit the technical contributions.
- Section 4.2: how do you learn the inducing points? The method in this section uses inducing points as the parameters for information sharing among different clients to overcome the nonparametric nature of GPs, so the paper [1] listed below should be cited here which uses another common technique (random Fourier features) for the same purpose.
- Section 4.3: lines 208-209, the inducing points are also defined globally by the server, so it's unclear to me what's the difference between the methods in Sections 4.2 and 4.3?
- Section 5: As mentioned above, the results in this section seem to have nothing to do with FL. Therefore, I doubt whether it will add much value to the current work as a paper for PFL.
- Section 5: just to clarify, is it correct that Theorem 1 is an existing result from [56], and your contribution here is to calculate the upper bound for your specific choice of GP classifier with Polya-Gamma augmentation? Lines 248-249: why have you chosen this approximation for $Q(\boldsymbol{\omega}_i)$? Is it like a marginalization over the $\boldsymbol{f}_i$'s?
- Experiments, is there a way to quantify and compare your computational cost with other methods? Because GPs are usually associated with large computational cost compared with parametric models like neural networks. One of the baselines is "Local" with uses a GP together with a locally learned kernel network, so have you tested the performance of standard GP (without kernel learning)? Not sure if I understood correctly, in line 292, it is mentioned that for CIFAR-10, every agent only has access to data belonging to two classes; however, in Table 1, how can the Local method still achieve an accuracy of above 70%? The same applies to the other two datasets.
- The paragraph starting from line 127: I feel that more technical background on GP-Tree is needed here since it's the backbone for all algorithms in this work.

[1] Federated Bayesian optimization via Thompson sampling. Dai, et. al. NeurIPS 2020.

Some minor comments:
- line 119: one of the $\mathbf{x}_i$'s inside exp should be $\mathbf{x}_j$
- line 269: "federate" -> "federated"
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work considers the task of learning user personalised predictive models in the federated learning setting. It is realised via a local Gaussian Process (GP) on each client which operates on top of features extracted from a neural network that is shared across the entire federation, named pFedGP. The authors motivate the usage of GPs via their Bayesian nature which allows for good performance in the low data regime (common in federated learning) as well as the extraction of prediction probabilities that are calibrated (crucial for safety-critical applications of FL). 

Since the authors operate on federated classification tasks,  they employ the GP classification model proposed at [1], which decomposes a multi-class prediction problem into a series of binary predictions. The benefit of this is that one can use the Polya-Gamma augmentation scheme, which can allow for efficient sampling of the posterior distributions, necessary for learning the hyper parameters of the model. The authors then  introduce two additional variants of their overall model which employ inducing points, a common technique in the GP literature. The first one is motivated for improving the performance of the model in the case of limited data, whereas the second one is about making the model more scalable in the case of large local datasets. Finally, the authors also provide a PAC-Bayesian generalization bound for the model performance one new clients that enter the federation. 

The paper concludes with a series of experiments that compare the performance of pFedGP against several baselines for federated learning and personalised federated learning. 

[1] GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning, Achituve et al., 2021
main_review: This work tackles the problem of personalisation in federated learning which is an important and active area of research in federated learning. The parts of the proposed approach are not novel themselves, however their combination for the task of personalised FL is new. The discussion of the method is relatively clear, and it is nice to see that the uncertainties provided by pFedGP are well calibrated. 

Despite these positive aspects, I cannot recommend for accepting the paper as is currently and below I elaborate why:
- The overall method seems quite computationally intensive, as one needs to perform clustering and several iterations of Gibbs sampling in order to learn the model. Since the compute is done locally in resource constrained devices in federated learning, I believe it is important for the authors to contrast the computational requirements (e.g., in terms of floating point operations) of their method against the more traditional NN training of, e.g., FedAvg. Furthermore, it would also be helpful if the authors include a pseudo-code of the overall federated training procedure with the different variants of pFedGP. 
- The generalisation bound provided in the submission is independent of a core part of the model, the neural net. As a result, one can wonder about its usefulness in practice where a wide variety of neural networks might be employed. For the current experiments it seems to work reasonably well, however the network used is a simple LeNet and the datasets (CIFAR10/100, CINIC-10) are quite  for the complex for this architecture (so it is harder to overfit). In an ideal case, it would be good if the authors include the neural network in the generalisation bound. As this might be harder to do, it would perhaps be worthwhile if the authors can show the behaviour of the bound with more complicated networks that have more parameters. Furthermore, there are some approximations involved in computing the bound with the Polya-Gamma augmentation (i.e., the expectation over $Q(w)$ and estimation of the mutual information). It is a bit unclear how finite-sample estimates of these quantities affect the actual bound and, ideally, they should be taken into account when providing the actual guarantee. 
- The experimental section and setup is a bit unclear with respect to a couple of important FL training hyperparameters. The authors use non-iid settings for the experiments. In prior work [2], it has been shown that in such scenarios, doing less epochs of local optimization (denoted as $E$) and using more clients per round can be beneficial for the final performance of the model when using FedAvg (which makes sense, as more local epochs can push each of the local models further away from each other, as each client has its own specialised dataset). Going over the authors’ appendix, it seems that for the baselines they use $E \approx 20$ but $E \in set(1, 3)$ for pFedGP. In fact, the reference [2] has managed to train a Lenet type of network with FedAvg on a similar non-iid version of CIFAR 10  and got up to ~73% performance, which is much better than the ~60% this work reports. It would be good if the authors adapt the hyperparameters appropriately so that the comparison is more meaningful. Finally, for a better personalised model performance baseline that is quite simple you should consider the performance of the locally fine-tuned model when training with FedAvg, see [3].
- One result that was relatively surprising was that FedPer had much worse performance than pFedGP, as they are quite similar in a sense; they both use the same backbone, except that one uses a linear classifier whereas the other uses a GP. It would be great if the authors can explain why this is the case. Furthermore, upon inspecting the author’s code it seems that they use two convolutional followed by two fully connected layers + one extra embedding layer, before feeding the embeddings to the GP. This is a larger model than the traditional lenet which, after the two convolutional layers, uses a single fully connected  layer and then the linear classifier. What kind of backbone did the authors use for their baselines? Did it have the same number of layers? (Unfortunately, I couldn’t find the code for the baselines in the supplementary). I would suggest that, for future comparisons, the authors, use a comparable architecture to lenet, in that they use the output of the first fully connected layer as an embedding for the GP.

Finally, below I have some further comments about the clarity of this work and how it could be improved:
- At line 192 in the main text the authors mention that inducing points can distort the true class distribution for pFedGP and that they use reweighting to combat this effect. In order to improve the clarity of the submission, I believe that the authors should provide an example of this event and why it happens. Furthermore, it would be beneficial if they explain how critical is this modification and how it is done in practice (i.e., what is $q(y, x)$, $p(y, x)$?).
- On lines 301-302 the authors mention that pFedGP-data is especially helpful when few datapoints per class are available, nevertheless, this seems to be the case only for CIFAR100 but not for CIFAR10 / CINIC-10. It would be helpful in the authors explain why. Finally, it seems that the inducing points were fixed to 100 for all the experiments? How critical of a hyper parameter is the number of inducing points for the performance of pFedGP?


[2] Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification, Hsu et al., 2019

[3] Improving Federated Learning Personalisation via Model Agnostic Meta Learning, Jiang et al., 2019
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new framework for personalized federated learning via Gaussian process classification (GPC) model. The key idea is to have each client node maintain its own GPC model but share the NN feature map f(x). Here, the GPC model of client c is equipped with kernel k_c(x, x') = k_c( f(x), f(x'); w_c)  with personalized parameters w_c. 

Thus, per iteration, a subset of clients is selected. Each client c downloads the shared f(x) from server and re-build its GPC model (i.e., re-fitting for w_c) using local data D_c. A local updated feature map f(x) (via gradient propagation) is then computed for each client and communicated to the server, which averages the updated model to form a new feature map f(x).

The proposed framework is evaluated on 3 image datasets: CIFAR-10, CIFAR-100 and CINIC-10. The reported performance shows that it outperforms a selected set of baselines substantially when the amount of data per client is small & heterogeneous.
main_review: NOVELTY & SIGNIFICANCE

I believe this is the first personalized federated learning framework that emphasizes on separating between learning the model (which is private) and learning feature representation (which is shared and can be learned via FL). This is an inherent separation the construction of a GP model which makes it a good model base. In addition, choosing GP as a base model is also a practical choice seeing that GP is often more effective on low-data regime (thus, being more suitable to provide personalized performance to client with little data). Thus, concept-wise, I consider the proposed method novel. 

On the other hand, the proposed method appears to combine straight-forwardly a bunch of existing techniques (i.e., a vanilla local GP training method, direct adoption of FedAvg to learn a shared feature map & a vanilla PAC bound to assess the generalized performance on novel clients) so algorithmic-wise, there is little novelty & significance. That being said, I also believe the significance of the proposed method is more on the practical side, regarding its extensive empirical studies, on which I will further comment below.

QUALITY

The paper is technically correct. I have gone through its technical development & derivation and have not found any flaws. I do, however, find the contributions of the paper somewhat orthogonal at certain parts: 

First, it is not clear to me the role of a GP-tree model in this context -- I thought any GPC would do so if we are to pick a more sophisticated model, there must be a reason. I guess it has to do with the performance but I am not sure why so could the author elaborate more on this? 

Likewise, I also do not see the advantage of using Polya-Gamma augmentation over normal approximation techniques for GPC. Please also discuss this in the rebuttal.

---

Regarding the experiments, I am a bit confused that pFedGP-IP-data is inferior to pFedGP-IP-compute. This is somewhat strange since pFedGP-IP-compute only uses the shared inducing points so there is not much to personalize. In contrast, pFedGP-IP-data includes both inducing points and local data points for inference so it is not clear to me why pFedGP-IP-data is inferior to pFedGP-IP-compute. Could the authors elaborate more on this too?

In addition, I find the personalized FL work in [16] very relevant & is directly applicable to the setting here. Could the author provide extra, minimal comparison with [16] in the rebuttal?

---

Regarding the theory, I suppose the point of section 5 is that if we use the feature map directly to build a personalized GPC to a novel client, we can make use of the PAC-bound for GPC in [56] to (empirically) measure the generalization gap. While this is not exactly a novel point, I appreciate the empirical analysis that makes use of this bound to showcase the enhanced OOD generalizability of the proposed model on novel clients, especially there is an increasing distributional shift. This speaks well to the practical significance of the proposed method.

CLARITY

The paper is very well-written. However, I feel that to a more general audience, the lack of background cover on Polya-gamma & GP-tree might impair readability. I'd suggest cutting down on section 5 to make room for those background detail. 

REVIEW SUMMARY

In general, I enjoy reading this paper. The proposed method provides quite a refreshing perspective and it has also been evaluated meticulously. That being said, I do have some questions for the authors & some of which is rather important (e.g., comparison with [16]) and I hope the authors would address those in the rebuttal.
limitations_and_societal_impact: The authors have included a broader impact statement elaborating on the societal impact of the proposed work. I find that acceptable & have no further concerns.
review_title: null

Meta-review: The paper adopts and adapts a number of known GP learning techniques to develop a method for personalised federated learning. The proposed method is computationally intensive, but performs well in practice and even comes with theoretical utility analysis. After author feedback, all reviewers appear happy with the paper and recommend acceptance, assuming the authors revise the paper to incorporate the material from the author feedback.
Decision: Accept (Poster)