Venue: NeurIPS 2022 Workshop Federated_Learning

Reviews:

final_score: rating: null
review_content: *Summary*: This work focuses on personalized FL by proposing a server-less framework called FedeRiCo where each client decides which collaborators to use for personalization by using the expectation-maximization algorithm. The authors also provide convergence guarantees for FedeRiCo and also propose a communication-efficient version of FedeRico. With three different datasets, the authors show that FedeRico achieves higher personalized test. accuracies across clients than other standard FL and personalized FL baselines. 

*Strength*:
- The proposed algorithm FedeRiCo, with convergence guarantees, can achieve good personalized FL performance in a decentralized (server-less) setting.
- FedeRiCo also has a communication-efficient protocol to lift the burden of the cost of system resources.

*Weaknesses*:
- FedeRiCO can inherently be prone to adversarial attacks if malicious clients participate in the framework.
- The computational cost for performing the EM algorithm on each client should be evaluated. In general, clients may have limited resources and performing the EM algorithm as compared to local SGD steps in standard FL can have a higher computational cost. 


review_title: Review

final_score: rating: null
review_content: The paper proposes FedeRiCo, a decentralized FL framework wherein each client finds the "right collaborators" from the entire pool of clients. The authors provide convergence guarantees and demonstrate empirically that FedeRiCo outperforms other standard FL baselines. 

**Pros**

* The idea is well-motivated and emerges naturally from the graphical model formulation used by the authors in Section 3.1. In some aspect, it is similar to "soft K-Means" where instead of grouping clients into hard clusters, we do a "soft-clustering" based on a loss-based similarity metric between all clients.

* Experimental results looks promising and show that the proposed framework outperforms ClusteredFL as well as other personalized approaches like FedEM and FedFOMO. 

**Cons**

* **Assumptions:** The assumption that $p_i(x) = p(x)$ in Line 70 seems unnatural. It is not satisfied in most cases and even in the experiments that the authors simulate. Clearly, the data at each client comes from different distributions and therefore this assumption cannot hold. I would suggest the authors to relax this assumption in the main paper itself (as they claim is possible in Line 71).

* **Local SGD Steps:** The biggest drawback for me is that the authors make no effort to incorporate local steps either in their problem formulation or even in the  practical solver. It has been well-established in FL that clients perform multiple local SGD steps before communicating with the server or their neighbors in the decentralized setup (see [1]) in order to reduce the communication efficiency. Showing convergence guarantees with local steps is the challenging aspect in FL; otherwise it just reduces to distributed GD which is easy to analyze. 

* **Communication Cost:**  The communication cost at each client scales with the total number of clients in the FL system. Specifically, if we let $K$ be the total number of clients and $d$ to be the dimension of the model, then the upload and download cost at each client is $\mathcal{O}(Kd)$. This is the price you have to pay for "soft-clustering". In hard clustering, the communication cost only scales with the number of clusters $c$ (see [2]). The authors propose a heuristic client sampling scheme to deal with this but this has its own concerns as I mention below.

* **Difficulty in understanding client sampling:** Firstly I would suggest the authors to describe the communication efficient protocol in Section 2.2 in an algorithm environment. Given the several heuristic relaxations it becomes difficult to keep track of the final algorithm. My main concern is difficulty in understanding how the client sampling procedure fits into FedeRiCo. Suppose client $i$ samples the set of clients $\mathcal{S}_i^{(t)}$ at round $t$. How are the updates computed for clients that are not sampled, that is $j \notin \mathcal{S}_i^{(t)}$?  Do you set the update to be zero for such clients or do you reuse the update received the last time the client was sampled? It would also be good to provide convergence guarantees with this client sampling procedure.

References

[1] Koloskova, Anastasia, et al. "A unified theory of decentralized sgd with changing topology and local updates." International Conference on Machine Learning. PMLR, 2020.

[2] Ghosh, Avishek, et al. "An efficient framework for clustered federated learning." Advances in Neural Information Processing Systems 33 (2020): 19586-19597.


review_title: Interesting approach but several practical concerns

Decision: Accept (Poster)