Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Needs Clarification (NC): a submission which does not meet the reviewing criteria and needs clarification for its described problem or solution
review_content: summary_of_contributions: This paper proposes a peer-review mechanism for federated learning, in which multiple reviewing clients are selected for each client. During the training process, each client sends its local model to the reviewers and receives scores. At the end of the training, each client chooses the local model with the highest score as its final local model.
strengths_and_weaknesses: **Strengths**
- The paper proposes an interesting algorithm for federated learning.
- The submission of this paper meets the formatting requirements and page limits.

**Weaknesses**
- The paper claims that existing methods only focus on the performance of in-house distribution and neglect the generalization ability of the model for the overall problem. However, this claim is inaccurate, as most of the non-iid methods in the literature report their accuracy on a global test set.
- The description of the method is too vague, and only the selection criterion for the review clients is given in the main text.
- The proposed method simply selects the best-performing model among those obtained from multiple rounds as the final local model. I do not think this method has any value for publication.
- The algorithm description in the appendix seems to be missing a key part, which is the interaction process with the central server. Is this a decentralized framework?
- The paper points out that current research only focuses on small or simulated datasets, yet the experiments in this paper are also conducted on small datasets.
- The paper does not provide any experimental results for other non-iid methods.
suggested_changes: - The authors need to provide a detailed description of their method in the main text. Currently, the main text only describes how they selected the review clients.
- They need to add comparative experiments with other non-iid methods. There are no experimental results from any relevant non-iid papers for comparison.
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: The article considers the impact of Federated Learning on the individual client's performance for their own local data (in-house distribution) and general unseen but similar data (generalization). The authors propose a solution that can optimize for both in-house and general distributions unlike prior methods.
strengths_and_weaknesses: Strengths:
- The article investigates an interesting problem of improving both the in-house local distribution and generalization performance of local clients in a Federated Learning (FL) Scenario. The authors suggest that after FL convergence, each client can locally train models (local adaptation) and then submit its model to nearby clients for improving their generalization performance. By optimizing over a weighted generalization and personalization loss, the clients are better prepared for both kinds of problems.
- The experiments show significant improvement in the clients' generalization performance when they are trained with the proposed Local and Global Federated Learning (LFGL) algorithm rather than the basic local adaptation algorithm.

Weaknesses:

- Experiments comparing other baselines in this space can be useful to assess the importance of these results.
- Privacy and communication costs of LGFL due to model sharing over clients are not studied. Intuitively, both of these might be on the higher end.
- Results that show the possible decrease in personalization performance due to LGFL are missing.


suggested_changes: - For the Local and Global Federated Learning (LFGL) algorithm, it is unclear how the peer-review process proceeds when the clients are not selected by the Client Election (CE) algorithm. Furthermore, the privacy implications of directly sharing models trained on local data are unclear and unstudied. 
- From the results, it looks like the LGFL algorithm primarily contributes to the strong combined generalization and personalization gains. Therefore, it is unclear if the CE algorithm which is also costly and scales quadratically with the number of clients is useful. Similar to LGFL, the privacy implications of finding similarity metrics for clients are unstudied. Adding these extra details can improve the contribution majorly.
- Theoretical basis and intuition for cross-sharing of models should be provided. For example, the number of peer-reviewing clients or rounds of training required beyond local adaptation for convergence needs to be studied.
- A minor comment for readability: Since the LGFL algorithm alone shows higher generalization gains, it might be useful to shift this algorithm in the paper's main section. Shifting the Client Election algorithm to the appendix should be okay. Furthermore, the article writing and definitions can be considerably improved.
review_title: null

Decision: Invite to archive