Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work considers a new personalized federated learning scenario where this federated training needs to learn some intrinsic model property and generalize this knowledge into new client. This considered scenario is interesting. To overcome it, this work attempts to construct the shareable model bases which each client utilizes to learn combination coefficient. In addition, this work evaluates the effectiveness of the proposed method on datasets with domain-shift.
strength_and_weaknesses: Strength: 

1. The considered new PFL scenario is interesting and practical.

Weaknesses: 

1. Some details of the proposed method are not clear.

2. The experimental settings are not convincing.

clarity,_quality,_novelty_and_reproducibility: The writing part is fine, and the proposed method is a litter new in PFL area.
summary_of_the_review: This work considers a new personalized federated learning scenario where this federated training needs to learn some intrinsic model property and generalize this knowledge into new client. This considered scenario is interesting. To overcome it, this work attempts to construct the shareable model bases which each client utilizes to learn combination coefficient. In addition, this work evaluates the effectiveness of the proposed method on datasets with domain-shift.

However, I also have several concerns on this work:

1. How to learn and select the representative model bases is not clear? 

2. When the new clients come from new domain or other new distributions, I wonder if the proposed method also can solve it. In addition, the idea of generalizing model into new clients (domains) is very similar with domain generalization (DG). Thus, it would be better to discuss the association and difference with some DG works.

3. This work only conducts experimental comparisons with some baselines on new data setup, which will be not convincing. Thus, it would be better to add some experiments as the traditional PFL works (e.g. FedRep) on Cifar-10/100 or Fe-MNIST.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: First, this paper propose a personalized federated learning (PFL) method, FEDBASIS, based on coordinate descent trying to learn a set of bases for the overall system and personalized coefficients in federated learning setting. Second, a benchmark for PFL, PFLBED, is proposed discussing the procedures to build non-i.i.d. of joint distributions and to maintain reliable evaluation simultaneously.
strength_and_weaknesses: Strengths:
[+] An architecture and algorithm, FedBasis, is proposed to decouple ‘the required total number of personalized parameters from the number of clients’.
[+] Novel benchmark for PFL: the motivation is worth considering.
[+] Remarkable experiments’ results.

Weaknesses:
[-] It seems like that the straightforward ideas of FedBasis are not novel and fail to decouple the personalized parameters from the number of clients in practice due to the requirement of the set of basis and calculation of the gradient based on them during training (of the set of basis even during inference, ). Can you provide more explanation about decoupling rather than duplicating parameters with more copies?
[-] The procedures of benchmark are not comparing to existing benchmark (e.g. LEAF) experimentally on discrepancy as you claim in Section 5. Can we employ them on any classic ML dataset?
[-] What about test on LEAF benchmark? And the results are relevant to benchmarks or not?
[-] FedBasis is claimed to be not sensitive to learning rate and training epoch, but Per-FedAvg is. How about to well-tune two versions of Per-FedAvg, FO and HF.
[-] The poor experiments about robustness of personalization lead to low persuasiveness.Why not consider comparing methods to reduce structural risk and to use regularization to deal with it, such as pFedMe, since the clients may not have enough data for validation in practice?
[-] The accuracy, loss, etc. of these training processes are not shown but only tables of acc.

clarity,_quality,_novelty_and_reproducibility: This article is very readable and has some novel ideas while being very easy to reproduce.
summary_of_the_review: The ideas of FedBasis are not novel but the motivations of benchmarks are of interest. Some experiments are not solid enough to support their claims.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new method for personalized federated learning. The key idea is to represent neural network weights as a linear combination of multiple bases, where the bases are shared across clients and only coefficients are personalized. Doing so makes it easier to personalize the model for a new client who did not participate in the training, by just training the coefficient parts. A coordinate decent approach is introduce to train the both of personalized coefficients and shared bases in an alternate fashion. Experimental results on a new benchmark dataset shows the effectiveness of the proposed method over a variety of existing methods.
strength_and_weaknesses: 
## Strong points
- Overall, the paper is well written, structured, and easy to read.
- The proposed work is well motivated. I agree that learning to personalize a learned model for a new client who was absent in the training is an important and interesting topic of personalized FL.
- The proposed method is designed reasonably and technically solid. While learning to linearly combine multiple kernels is a known idea (Yang et al., 2019; Chen et al., 2020; Zhang et al., 2021c), adopting it to personalized FL is new as far as I have checked.
- A new benchmark dataset (more precisely, data splitting protocol) is introduced for personalized FL.

## Weak points
- While I believe that the proposed method seems to have some technical merits, the current experimental results only demonstrate that "the proposed method worked better than existing methods on the proposed datasets", which is a bit less convincing to me. Overall, it is not very clear what motivates to develop a new benchmark dataset, though I appreciate the introduction of new dataset itself. In the paper, it was argued in the introduction that the existing PFL evaluations are (1) misleading due to the mismatches between training/test distributions and (2) less comprehensive because the non-iidness arises only in either labels or input domains, not both. While I agree with (2), I would think that train/test mismatches in (1) could happen in practice as well studied in domain adaptation and generalization. The paper could have been stronger if the proposed method is compared with baselines also on other datasets (i.e., data splitting scheme) used in prior works. 
- It remains not obvious if the proposed method really outperforms baselines from the current results. In Table 2, the performance gains from the most strong baseline (Per-FedAvg + FT) are less than 3 percent points in most cases, on average. How about the variance of test accuracy across clients and multiple runs with different random seeds? Such additional information is necessary to fully demonstrate that the performance improvements by the proposed method are statistically significant. 
clarity,_quality,_novelty_and_reproducibility: - **Clarity**: The paper as well as its contribution is presented clearly. 
- **Quality**: The quality could be improved a bit more. Specifically, I have some major concerns regarding the experimental evaluation (commented above).
- **Novelty**: The paper has some technical novelty. While learning to linearly combine multiple kernels is a known technique, adopting it for personalized FL seems new.
- **Reproducibility**: The current submission has limited reproducibility. The only pseudo code is provided for the proposed method. It is not trivial to fully reproduce the results as doing so requires full replication of data splitting and training procedures with certain random seeds. 
summary_of_the_review: Overall the paper is well written and presenting some technical contributions for personalized FL. Nevertheless, I am currently neutral about this submission due to its limited experimental evaluation. If the proposed method is confirmed to (1) statistically outperform baseline methods (with standard deviations or confidence intervals regarding multiple clients and multiple random seeds) and (2) work effectively also for other datasets/data splitting scheme used in prior work (e.g., those used in pFedHN), I would be willing to upgrade my score. Code submission for a complete reproduction of this work is also highly welcome.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission proposes a strategy for training a set of base models, a linear combination of which will be taken for each client, as a function class for personalized federated learning.  The main learning algorithm is highlighted in Section 4.2, Eq (9).  Section 4.3 then provides arguments for why this might be theoretically justified.  The submission received a range of reviews, with one reviewer being convinced by the rebuttal arguments, and others being less responsive.  Some common concerns were with regard to technical novelty, and whether the resulting method is sufficiently theoretically grounded to make a lasting impact in the field of personalized federated learning.  Quoting from the authors' rebuttal
"Our technical novelty is about how to learn the bases successfully in a federated setting, which is non-trivial as discussed in Section 4.2 and shown in our experiments in Table 3 and Section 6. In other words, while the idea looks superficially straightforward, making it work is our key contribution."
Of key concern, then is whether the procedure, which is presented in a somewhat heuristic fashion, is sufficiently theoretically grounded.  This is contained in Section 4.3, which is 2 paragraphs and essentially repeats Theorem 1 from Shamsian et al. (2021), which appeared in ICML.  This shows the benefit of decoupling the number of parameters from the number of clients, but does not really resolve the heuristic nature of the procedure described in 4.2, which comes without guarantees, and differs from the objective described in Eq (7) in ways that are not in the end resolved and concisely described.

As a result, the submission describes a method that has some intuitive appeal and is an interesting empirical study, but in a topic that has a large number of results (34k hits when searching for personalized federated learning on google scholar) but is in need of frameworks with a proper theoretical grounding and justified learning procedures.
justification_for_why_not_higher_score: The submission's main contributions boil down to the procedure described in Sec 4.2, but this is presented in an imprecise way that does not justify the procedure from a theoretical basis or a concisely described learning objective.  Theoretical grounding is not provided further than a result from ICML 2021 indicating that decoupling model complexity from the number of clients can be beneficial.
Decision: Reject