Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To overcome statistical heterogeneity among data shards in collaborative federated learning, this paper proposes a novel personalization schema which only requires smaller memory footprint in clients and possibly is less susceptible to catastrophic forgetting. The main idea is to split the trainable parameters into  shared and personalized parameters where, unlike existing personalization schema, only the shared model is exchanged with the server in communication rounds to be aggregated- thus partial personalization. In regression, this corresponds to learning the residual error of shared model via personalized model and in the classification setting corresponds to output averaging (unlike interpolation based personalization methods that do parameter mixing). The authors propose two algorithms FedSim (simultaneous updating of shared and personalized models locally) and FedAlt (alternative updating of shared and personalized models locally), to learn in this setting, theoretically analyze the convergence rates in non-convex settings and conduct empirical studies on image classification and next-word prediction to evaluate the proposed methods.

main_review: This paper studies an interesting question and proposed idea looks interesting with corroborating empirical results, but there are few issues that prevents me from giving it a high score:

- The key motivation for the paper is to utilize a small footprint in clients but the proposal is somehow mis-leading. I understand the learned personalized model is significantly smaller than the shared model (e.g. 1-2% as observed in experiments for some applications) and requires a small footprint after deployment (inference stage), but during the training the clients need enough memory for both models to participate in collaborative training. Also, I was left wondering how large the coupling between personalized and shared parameters captured by $\chi$ would be in presence of heterogeneity and significant difference in model sizes. 

- While the proof of theoretical results looks sound as far as I checked, the obtained rates are hard to interpret and poorly elaborated. For example, the gradient diversity assumption (Assumption 3) is only defined over the shared model. Also, the role of gradient diversity term, \rho, which appears in tuning the learning rate $\eta$ is ignored in discussions (e.g. Table 1) which might dominate other terms if incorporated properly. So, it seems hard to put the obtained rates in the context of known results.

summary_of_the_review: This paper proposes a solution for personalization in federated learning where clients only exchange a small portion of parameters with a server for collaborative learning. The problem and proposal looks interesting, but it would be great to hear from authors on some issues raised above before final evaluation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies personalization in federated setting, i.e., instead of collaboratively training a global model, personalizing a model for each client.  This paper proposed to personalize only part of the model parameters instead of the full model, and studied two algorithms FedSim and FedAlt. In local client updates, FedSim will simultaneously train the shared and personalized parameters, while FedAlt will train the personalized model first, then train the shared parameter. 
main_review: This is a well written paper and easy to follow. However, I have the following concerns. 

An important related work is missing, and I strongly suggest the authors discuss
- [Singhal et al. 2021 Federated Reconstruction: Partially Local Federated Learning https://arxiv.org/abs/2102.03448]. If I understand correctly, the proposed FedAlt algorithm is very similar to the paper, except that FedAlt requires a local state v_i. 

Clarification on contributions
If I understand correctly, FedSim has been used in [Liang et al. 2019, Arivazhagan et al. 2019, Collins et al. 2021, Li et al. 2021 ], FedAlt is relatively new but closely related to [Singhal et al. 2021]. This paper makes the following contributions
- Convergence guarantees of FedSim and FedAlt: standard convergence rate under reasonable assumptions.  Li et al. 2021 presented a convergence rate of FedSim, could the authors compare?  Regarding the novelty of FedAlt proof, I believe virtual proxy variable is a common technique in federated optimization, (e.g., eq(13) in [Wang et al. 2021 A Field Guide to Federated Optimization https://arxiv.org/pdf/2107.06917.pdf]), and I would appreciate it if the authors elaborate more on the novelty. 
- FedAlt algorithm, which is a client state variant of Singhal et al. 2021, and achieves marginal improvement over FedSim [Liang et al. 2019, Arivazhagan et al. 2019, Collins et al. 2021, Li et al. 2021] (always < 0.5%.)
- Theoretical insights on when FedAlt is better than FedSim. I found this to be interesting, and I would encourage the authors to provide more discussion and connections to empirical results.


Savings of memory footprint
- It would be good to show either analytically or empirically how much memory the partial model training can reduce compared to full model training as this is the main motivation. 

Clarification on some details. 
-  Authors mentioned “All methods, including FedSim, FedAlt and the baselines are initialized with a global model trained with FedAvg. ”: how is the global model trained? And is it trained on the same set of clients as personalization?
- For table 3, does the full model in baselines include adapter parameters? 
- How do the authors split the dataset for training, personalization and testing?

summary_of_the_review: I strongly encourage the authors clarify contributions explicitly acknowledging previous works [Liang et al. 2019, Arivazhagan et al. 2019, Collins et al. 2021, Li et al. 2021 ], as well as  [Singhal et al. 2021 Federated Reconstruction: Partially Local Federated Learning https://arxiv.org/abs/2102.03448].
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a personalized FL framework with partial model personalization. It separates the model parameters into two parts, shared model and personalized model, and optimize them in an interleaving manner. The authors proposed two optimization algorithm, named as FedSim and FedAlt, with partial clients participation. They also analyzed the algorithms' convergence rate on general smooth nonconvex function. In experiments, they consider 3 different model split methods: 1. input layer as personalized model, the rest as shared model; 2. output layer as personalized model, the rest as shared model; 3. adding adapter as personalized model.  The experiments conduct on NLP or vision tasks also demonstrate that the proposed algorithm outperforms other personalization method.
main_review: Pros:
1. The authors proposed a unified framework for personalized federated learning. To my best knowledge, they give the first analysis of this framework and algorithm on smooth nonconvex function.
2. The proposed algorithm can outperform existing personalization algorithms on NLP or Vision tasks.

Cons:
1. The personalized layer idea is not new, as authors mentioned, it has been proposed in prior works [Arivazhagan et al. (2019) and Collins et al. (2021)].
2. I think the established theory does not perfectly fit with experiment model, since they assume smoothness in theory, but in practice the motivated example is usually non-smooth ReLU network. Hence I think it will be great if they can perform analysis on ReLU network, perhaps very simple one layer setting.


summary_of_the_review: Overall, this framework is interesting and theory is new. However, the novelty for me is somewhat limited. If they can come up with theory that can explain their empirical observations (ReLU network), I believe it will be a strong work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper discusses using partial personalization objective function defined in equation (3) to achieve personalized models in federated learning. While the idea of partial personalization and the objective of (3) have been widely studied in previous literatures, the main contributions of the paper are the convergence analysis of two proposed algorithms FedSim and FedAlt in nonconvex case. The authors also did extensive experiments to compare partial personalized models with fully personalized models by using various model structures and on different datasets.
main_review: My main concern about the paper is the limited novelty. The idea of partial personalization has been proposed in previous literature. The objective function (3) has been detailly studied in [1]. Moreover, the convergence of FedSim for nonconvex case has also been studied in [1] under very similar assumptions (See Section 3.1 of [1], where the name of FedSim is LSGD-PFL). Thus, the only new result left is the analysis of FedAlt under nonconvex setting. In order to derive Theorem 2, the authors claim that they proposed a new technique called virtual full participation; however, this technique is actually not new and has been used widely in convergence analysis for FedAvg/Local-SGD. This way, the whole contribution of the paper relies on the analysis of FedAlt in nonconvex case based on well-known techniques, which seems quite weak contribution to me. I would recommend the authors to give more detailed explanation on what the key theoretical challenge in deriving Theorem 2 is. 

The extensive experiments of using different model structures on different datastes is appreciated. And the observation personalization can hurt the test accuracy on some devices is interesting. But given the idea of partial personalization is proposed elsewhere, the gain from the experiments is limited.

The writing is clear. I can easily understand the paper. Thanks for that.

In a word, given the ideas used in the paper are now new, I think this paper is maybe better written in a pure theoretical one. The key would be to stress the theoretical challenge and novelty. Based on the current version, I believe the contribution is not novel enough or at least it is not stated clearly.

[1] Filip Hanzely, Boxin Zhao, and Mladen Kolar. Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques. arXiv Preprint, 2021.
summary_of_the_review: The contribution is unclear, which constitutes strong reason for not accepting the paper.
review_title: null

Meta-review: Dear authors,

I have read the reviewers and your careful rebuttals. I would have liked to see much more engagement from the reviewers. However, even after your rebuttal, no reviewer suggested acceptance, with two reviewers proposing reject (3) and two proposing weak reject (5).

The reviewers found the paper well written. I concur. The reviewers also notice that the contributions are very marginal compared to prior literature. Personalized FL formulation studied here was in a simpler form first proposed by Hanzely and Richtarik (Federated learning of a mixture of global and local models, 2020) and later generalized by Hanzely et al - a paper the authors cite. That work performed an in-depth analysis, also including the nonconvex case, which the authors (claim) did not notice. Compared to that work, the authors perform an analysis in the partial participation regime. However, partial participation is by now a standard technique which can usually be combined with other techniques without much difficulty. The authors tried to argue that their analysis approach is unique, but the reviewers remained unconvinced. 

In summary, I think this is a solid piece of work which is perhaps judged, looking at the raw scores, a bit too harshly. However, most verbal comments are indeed fair. I am also of the opinion that the paper in its current form does not reach the necessary bar for acceptance. I would encourage the authors to carefully revise the manuscript, taking into account all feedback that they find useful. I think the paper can be improved, with not too much effort perhaps, to a state in which the bar could be reached. 

Kind regards,

Area chair
Decision: Reject