Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The work aims to address the problem of creating adequate personalized models in a federated learning regime. The problems stems from a general observation that global models do no generalize well in the Federated Learning (FL) scenarios. The authors propose to use the idea of knowledge distillation to properly transfer knowledge to different clients via soft prediction mechanics. It is based on the knowledge about predictions that can be obtained on a public dataset (also referred as collaborative knowledge). At the server side, the proposed model learns the corresponding knowledge transfer coefficients that align inputs from multiple clients and hence distill the knowledge. The training is performed in an alternating manner by switching between knowledge coefficients (updated at the server side) and the task-specific parameters of the model (updated at the client side). Advantages of the proposed approach are empirically demonstrated in comparison against other FL algorithms w.r.t. both accuracy and communication efficiency on several public datasets.
main_review: The text is well written and easy to read. It follows logical structure and is transparently organized. The related literature section is relevant and gives a good overview of the state of the field in this direction. The problem formulation and the corresponding derivations of the solution are mathematically sound and non-contradicting. The key concepts are properly introduced and clearly described. I find the way knowledge distillation utilized in this work very creative and novel. Considering that the FL topic draws a lot of attention from industry, the proposed approach seems to offer a very promising solution with high practical importance. It also open perspectives for further research in this direction, e.g. for finding better ways of aligning client data on a server side. I enjoyed reading this paper very much.

However, I also have several concerns and questions. 

1) In the very beginning, the authors mention privacy-related issues in the FL context. The proposed scheme promises to resolve any issues as it doesn't directly transfer any private user data in any way. On the other hand, it seems that in order for this model to work properly, the distribution of data on public and private datasets should be close (ideally the same). If so, then knowing distribution on the public dataset $\mathbb{D}_r$ may potentially help revealing distribution on the private data $\mathbb{D}_n$, which in turn would allow for some better-then-random-guess attempts to deanonimize user data. I'm not sure what are the risks of that and it would be good to have authors add more details on this issue and the general risks of loosing privacy in the proposed scheme.

2) Why is the total number of samples summed over all $D_n$ is equal to the number of clients $N$. It basically means that there's only 1 data sample per client. But clients may generate/use many data samples. Am I missing something?

*Typos*:  
- line 126: "to breaks the barrier"
- line 136: the $c$ variable should be in bold font, I guess
- line 147: "that larger", missing "is"
limitations_and_societal_impact: FL methods are traditionally connected with privacy issues. I have outlined my privacy-related questions in the main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper works on personalized federated learning (FL). The authors specifically studied the situation that clients train different model architectures. The authors leveraged additionally, public unlabeled data and knowledge distillation (KD) to aggregate these heterogenous clients' models specifically for each client, so that each client can learn their personalized model with the help from other clients. Experimental results on several datasets show the effectiveness of the proposed method.
main_review: I number my questions or the paper's weaknesses as follows. If possible, I would like the authors to at least address 1-4 and 7-8.

======= *Originality*: The proposed method is somewhat a novel combination of (1) KD for FL and (2) deriving a personalized global model/regularizer for FL. However, the authors did not explicitly state these relationships to existing works clearly. Instead, the authors mentioned some of the relationships distributedly in several places of the paper. Besides, many related works are not cited, discussed, or empirically compared. All these factors largely limit the papers’ originality.

1. There have been a few FL works on heterogeneous systems, e.g., [a, b]. Specifically, [b] has a title similar to the proposed method. Both papers are not cited. Also, there is no description in the related work on existing solutions to heterogeneous systems (e.g., [16]), giving a wrong impression that the proposed method is the first to handle this situation.

[a] HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients, ICLR 2021

[b] Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge, NeurIPS 2020  

2. Knowledge distillation (KD) is one promising way to handle system heterogeneity. However, instead of discussing how KD has been used in FL in the related work (e.g., [12-16]), the authors just provided a general overview of KD. Specifically, [13] is on personalized FL but without any discussion. Moreover, several other FL works using KD are not cited and discussed [c-f].

[c] FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning, ICLR 2021

[d] Distilled one-shot federated learning, arXiv 2020

[e] Federated Model Distillation with Noise-Free Differential Privacy, arXiv 2020

[f] One-Shot Federated Learning, arXiv 2019

3. Another important component is personalized knowledge transfer. It seems that [8, 24] share similar ideas to the proposed method, but the authors didn’t provide a detailed discussion. Some other similar works also derive personalized global models or regularizers [g, h].

[g] Personalized federated learning with first order model optimization, ICLR 2021

[h] Federated Mixture of Experts, OpenReview 2020

Without a clear discussion of these related works and a discussion on the challenge of a naïve combination of them, it is hard for me to assess the paper's originality.

======= *Quality*: The proposed method is technically sound. The optimization procedure is very clean and should be easily reproducible. The authors also conduct extensive experiments to justify the effectiveness of the proposed algorithm. However, there is no clear description and motivation on why it is designed. Also, some important baselines and experimental details are missing; some experimental designs can be improved. See the following.

4. More specifically, the authors suddenly gave a formulation in Eq (2) without giving the design principle. I would suggest that the authors first describe an objective function for personalized FL, and then gradually lead to Eq (2) for the heterogeneous case. Also, I don’t fully understand the description in Line 146-147 for the regularization term in Eq (4).

5. There seems to be no clear description of how the public data is constructed. Does each communication round use the same public data? Also, there is no study on the size of the public data (not merely the batch size).

6. Experimental designs. I checked the design of "each client contains all classes of samples" in the supplementary. It does not lead to a strong non-IID case. Please take a look at [c, i] and consider re-design the experiments

[i] Measuring the effects of non-identical data distribution for federated visual classification, arXiv 2019

7.	Experiments on heterogeneous systems. Can the authors provide more details on FedDF, which was not designed for personalized FL? Specifically, do the authors compare to the distilled model at the server or the local models after local update/fine-tuning? It is very important to compare to the local models. Can the authors compare to the original FedDF (and with local updates before evaluating on personalized data), instead of the modified version in footnote 2? Note that, FedDF does not add any regularized term like the one in Eq (2) in its local training. Also, [13] should be compared. These experiments are very important to ensure a fair and comprehensive comparison. Besides, can the authors further compare to a version of Eq (2) with identical c_m,n? That is, there is no personalized knowledge transfer. 

8.	Experiments on homogeneous systems. First, I don’t see a reason why the authors have to modify their algorithm for this setup (Line 249-252). Is this a limitation of the work? Second, the baselines in Table 2 are not very strong. Please consider comparing to some of [i, j, g]. Moreover, please compare to FedAvg after local fine-tuning [17, k]. Third, the proposed method has an obvious advantage --- the public data. Thus, I would suggest that the authors further compare methods based on KD (e.g., [13, 16, c]). For those not designed for personalized FL, the authors can simply fine-tune the global models to get personalized models. Finally, the accuracy of CIFAR-10 in Table 2 is fairly low.  Do the authors know why?

[i] Think locally, act globally: Federated learning with local and global representations, arXiv 2020

[j] Federated learning with personalization layers, arXiv 2019

[k] Salvaging Federated Learning by Local Adaptation, arXiv 2020

======= *Clarity*: The paper is well-written and organized.
Significance: The authors work on an important problem, federated learning under the heterogeneous system setting. I think the proposed method could serve as a strong baseline to be compared or built upon in the future. However, the significance of the paper is limited by the aforementioned questions.

**Post-rebuttal update**
I thank the authors for their tremendous efforts in addressing my concerns for multiple rounds. Many of my concerns are addressed so I increased the score from 4 to 6. I encourage the authors to follow my latest response to incorporate the feedback and their rebuttal into the final version.
limitations_and_societal_impact: The authors didn't discuss the limitation. However, this does not affect my score. I don't think the paper will have potential negative societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies personalized federated learning (pFL), where the local models have different structures and sizes. The proposed method enables each client to maintain a personalized soft prediction at the server-side. Empirical results show the strength of the proposed solution.
main_review: The paper proposes a knowledge coefficient matrix at the server side to guide the local training. The pFL is a timely topic and the proposed approach is also interesting. 

Originality: This work is a novel combination of well-known techniques.
Quality: The submission is technically sound, and all claims are well supported. 
Clarity: The submission is clearly written and well organized.
Significance: The others (researchers or practitioners) are likely to use the ideas or build on them.

Strengths: 
1.	Writing is clear. Details of each component are explained and reproducible from the appendix and the code.
2.	The methodology is reasonable. The proposed knowledge coefficient matrix is well-designed with intuitive justifications. 
3.	This paper presents non-trivial performance improvement against SOTA pFL methods on 3 popular benchmark datasets.

Weaknesses: 
1.	The introduction of the knowledge coefficient matrix may raise the concern of privacy leaks and the communication cost. 
2.	The use of the public data in step 2 in Figure 1 conflicts with the general FL setting.
3.	To my best knowledge, there are some other personalized FL works (Hierarchical Personalized Federated Learning for User Modeling, WWW'21; Exploiting Shared Representations for Personalized Federated, arXiv:2102.07078), which need to be listed as baselines.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a personalized group knowledge transfer training algorithm (KT-pFL) based on knowledge distillation that deals with model architecture heterogeneity in FL systems.

The authors claim the following contributions:
- This paper is the first to study personalized knowledge transfer in FL.
- Proposing the ‘knowledge coefficient matrix’ to identify the contribution from one client to others’ local training.
- Providing theoretical performance guarantee for KT-pFL and conduct extensive experiments.

main_review: **Pros/Cons**:

[+] The problem this paper aims to solve is novel,  to the best of my knowledge this is the first work to address model architecture heterogeneity in FL.

[+] The paper is well written and structured. 

[+] Good theoretical analysis on KT-pFL's performance.

[-] The usage of knowledge coefficient matrix $\textbf{c}$ makes this method unfeasible for large-scale FL systems. $\textbf{c} \in \mathbb{R}^{N \times N}$ where $N$ denotes the number of clients. Previous works in P/FL the Hub/Server transmits the model parameters which can be at extreme case billions of parameters. If we take a small federation size of $10K$ clients we will transmit $0.1$ billion parameters in each COM. The proposed approach will be effective (in terms of communication costs) only on small-scale FL systems. 

[-] A major limitation is the usage of public data and its size. In the experiments, the authors used $3000$ samples for their public data which for CIFAR-10 is $5%$ of the data. In Healthcare FL systems it would be hard to obtain such public data, especially in this size. The authors did not provide any discussion related to the public data and did not explain how it was partitioned. Did the baseline methods use these $3000$ samples or they are held out?
I also think that an important addition for this paper should be an experiment that investigates the effect of the public data size on performance.

[-] The authors did not address more recent approaches in PFL [1,2] and did not compare to them.

[-] The update rule of KT-pFL involves three update loops: (i) local training, (ii) distillation, (iii) c update.  This may lead to asynchronous between clients, in FL we aim to mitigate it as it costs time and resources. In addition, it will be interesting to check what is the clock time of KT-pFL compare to FedAvg.

[-] All the clients in the federation take part in each communication round - this is a big limitation for large-scale federations, in real life FL systems federations can contain millions of clients such that training this federation will cost lots of computational power and money. Other PFL methods sample a subset of clients at each COM round (most of them sample 5 clients per round). This limitation raises another question, have the authors run all methods on the same number of communications? each COM round in KT-pFL performs $N$ communications as $N$ denotes the number of clients, while for pFedMe and [1,2] less amount of communications occur per COM round. In FL we care about the number of communications it takes the model to converge.

[-] "the learning rate is set to $0.01$ for EMNIST and Fashion MNIST and $0.02$ for CIFAR-10." It is not clear if the mentioned learning rate values were fixed after HPs search or not and if these values were used for the baselines as well. From the text "All datasets are split randomly with 75% and 25% for training and testing, respectively" I understand that no HPs search was done and these values were used for the baselines as well.

[-] Intuitive baseline for the Hetro FL experiment would be to assign each group of clients (that have different model architecture) with a server and train them separately.

[-] Experiments were conducted on a limited number of clients (20), in real-life FL systems, we have thousands and millions of clients I encourage the authors to run experiments on larger federation sizes e.g. 500 clients.

[-] Missing information on homogenous FL experiment - it is not what "homogenous" refers to if it is only for the model architecture or to the data distribution as well. 

[-] The authors should provide STD analysis (i.e. running the experiments on multiple seeds).

**References**:

[1] Shamsian, Aviv, Aviv Navon, Ethan Fetaya, and Gal Chechik. "Personalized Federated Learning using Hypernetworks." arXiv preprint arXiv:2103.04628 (2021).

[2] Zhang, Michael, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. "Personalized Federated Learning with First Order Model Optimization." arXiv preprint arXiv:2012.08565 (2020).
limitations_and_societal_impact: The authors should discuss the limitations of their approach in addition to potential negative societal impact.
review_title: null

Meta-review: This paper proposes a novel approach for handling model architecture heterogeneity in federated learning. Theoretical analysis and numerical experiments demonstrate the efficacy of the new method. Good writing and clear structure also make the paper stand out. Based on the reviews, I recommend acceptance. Meanwhile, please make sure to incorporate the reviewers' comments and the rebuttal into the final version.
Decision: Accept (Poster)