Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a personalized federated learning (PFL) method to achieve communication efficiency, robustness and fairness, by projecting local models into a shared-and-fixed low-dimensional random subspace and using infimal convolution to control the deviation between the reference model and projected local models. Theoretical analysis about strongly convex objectives is provided. Experiments on both synthetic data and real data are used for evaluation. 
strengths_and_weaknesses: Strengths:
1. The studied problem about designing method to achieve communication efficiency, robustness and fairness in PFL is interesting and important.
2. The paper is well written with good organization.
3. Experiments show that the proposed method does achieve good performance on the adopted models and datasets.

Weaknesses:
1. The theoretical analysis is weak due to some ideal assumptions. First, only results for strongly convex objectives are provided. But most models in real practice are non-convex. Second, to analyze robustness and fairness (Section 4.2), the authors assume that the number of local update steps is infinite, which is unreasonable because in non-convex cases the algorithm will not converge if the number of local update steps is set to be infinite. 
2. The datasets are relatively small. 
3. The models, especially deep models, seem to be specially designed. Why not just adopt some widely used models like ResNet?

questions: 1. Does the method still converge for non-convex objectives?
2. How is the performance on widely used deep models like ResNet and larger datasets like ImageNet? 

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work aims to tackle three problems in FL: 1) communication efficiency, 2) robustness (Byzantine attacks) and 3) fairness. Specifically, this work projects local models into a shared-and-fixed low-dimensional random subspace and uses infimal convolution to control the deviation between the reference model and projected local models. Theoretically, it shows convergence on strongly convex objectives with square regularizers and mild dimension reduction. Numerical results are presented on synthetic data, MNIST, Cifar10 with MLP and CNN.
strengths_and_weaknesses: Strength:
1. The problems addressed by this work are very important.
2. Theoretically and empirically show the advantage vs. the existing methods
3. Extensive experiments are conducted, including comparison and different variants of experiment settings. 

Weaknesses:
1. The presentation of this paper is not easy to digest as multiple components are delivered without sufficient details and support. The authors also include "due to space limitation." I would suggest the authors consider the key messages to deliver in the conference paper. 
2. The theoretical results seem interesting. However, the assumptions (strong convexity and smoothness) are relatively strong and have a clear gap between the real deep neural network training scenarios. 
3. One related work could be included in the comperison:
Rothchild, D., Panda, A., Ullah, E., Ivkin, N., Stoica, I., Braverman, V., ... & Arora, R. (2020, November). Fetchsgd: Communication-efficient federated learning with sketching. In International Conference on Machine Learning (pp. 8253-8265). PMLR.

questions: 1. How do we generalize the theoretical results to neural network training (non-convex)?
2. What is the key technical novelty of using the proposed projection vs sketching-based efficient FL methods?
3. As the proposed methods seem easy to implement, would it be possible to test them on more complex tasks to make the methods more convincing?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper formulates the goal to balance different constraints of interest (i.e., communication efficiency, robustness and fairness) simultaneously. The authors propose a personalized federated learning algorithm named lp-proj, which is supported by theoretical analysis. Extensive experimental results show the effectiveness of the proposed method.
strengths_and_weaknesses: 1) This paper proposes a practical problem which has not been studied widely. 
2) The proposed method is technically sound and is well supported by theory.
3) Extensive experiments on three datasets show the effectiveness of the proposed algorithm. 

I also have some questions:
1) (Main concern) I have noted that the authors show the results of trade-off between average performance and performance fairness. Is there a trade-off between performance and communication efficiency / robustness? How to balance them?
2) It seems that this paper simulates Non-IID settings only based on the target label. Can this work be applied to other Non-IID settings, e.g. domain shifts.
3) Is the proposed algorithm robust enough to deal with emergencies, e.g., some clients drop out in the training stage? 

questions: Please refer to Weakness.
I will increase my score if the authors can address my questions.

limitations: 1) The theoretical analysis is based on the assumption that the objective function is strongly convex, which is very strong.
2) The fairness metric used in this paper is in terms of performance. I'm not sure whether it works well on other fairness metrics such as group fairness (DP, EO etc.).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of providing increasingly uniform performance across devices (fairness) through projection of local models into a shared low-dimensional subspace and infimal convolutions. The paper provides theoretical proofs for convergence and explores robustness and fairness using the proposed method. The effectiveness of this approach is demonstrated through extensive experiments on federated datasets, showing improved performance in personalization accuracy, communication efficiency, robustness and fairness.
strengths_and_weaknesses: **Strengths**:
- The paper is well-written and organized. Algorithms are easily understandable and the code is well-written.
- Thorough theoretical analysis with extensive proofs that demonstrate the convergence of `lp-proj`.
- Theoretical analysis and experiments support the method's claim to improve robustness and fairness.
- Extensive numerical experiments on strongly-convex objectives and heterogenous datasets demonstrate improved state-of-the-art personalization performance. The proposed method significantly reduces communication costs.

**Weaknesses**:
- Limited technical novelty with many of the ideas coming from `FedProx` and `pFedMe`. Note: extensive comparison analysis gives convincing evidence that the method is sufficiently different from previous work.
questions: - Why does `lp-proj` help reduce the variance of performance among clients?
- Why does infimal convolution help personalized performance of the local model?
limitations: - Limited focus on how the model improves personalization performance using this method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a Personalized FL method which aims to achieve robustness, fairness and communication efficiency simultaneously. 
They achieve fairness through personalization and regularization (i.e., $L^{p}$-regularization) similar to existing works , and propose low-dimensional random projection of the local client updates to improve communication cost. The experimental results show the improvement of these metrics compared to existing baselines. 
strengths_and_weaknesses: Strength
*  The paper appears to be theoretically sound as well as the claims seem to be supported by theoretical justifications as well as numerical experiments.
* The paper addresses three important challenges in Federated Learning Settings (Communication Efficiency, Fairness, Robustness) altogether, which seems to me that there is a gap in existing works. 

Weakness
* The Novelty seems to be a little low compared to existing works ,  the embedding of local updates to lower dimension seems to be main novelty here. 
* In terms of Personalization Accuracy Performance, the results are mixed between Ditto and new approach (Table 3 Appendix), however in the main body of the paper (i.e. Sec 5), it is claimed the new approach is better the competitors. 
* In terms of communication performance, lack of comparison with some existing works such as the one proposed here [1]. 
* No experiment or discussion around the robustness of the proposed approach vs other attacks such as Backdoor attacks [2] is mentioned in the paper. 

1. Federated Learning with Buffered Asynchronous Aggregation, 2022, Nguyen et al. 
2. Attack of the Tails: Yes, You Really Can Backdoor Federated Learning, NeurIPS 2020, Wang et al..
questions: The Robustness performance of Ditto for EMNIST dataset seems a little bit surprising to me, because the results reported in Ditto for FEMNIST does not suggest this poor robustness performance on EMNIST, can you elaborate on this ? have you performed hyper parameter fine-tuning for Ditto Algo ?

limitations: NA
review_title: null

Meta-review: metareview: The paper gives an approach to personalized federated learning, which incorporates client heterogeneity, robustness to attacks and fairness in performance. The proposed method is interesting, though the simultaneous incorporation of multiple concerns makes it somewhat difficult to disentangle the role of different choices. The authors present theoretical results and a fairly comprehensive empirical evaluation showing effectiveness of their method.

The accept decision is primarily based on the substantial empirical analysis. As mentioned above, various algorithmic considerations should ideally be studied somewhat in isolation, such as through ablations. For instance, it is completely reasonable to incorporate personalization through the infimal convolution without a subspace projection. Presumably the authors do not do this due to the computational cost, but the joint effect of P and the personalized formulation is a bit tricky for the reader.

For the theory, assumption 1(a) is effectively vacuous in most ML settings, where the loss \tilde{f}_k is typically based on one or a few examples only, and hence cannot be strongly convex without including additional regularization in the loss. I appreciate the addition of the non-convex result in the revision, and would advise changing this to the main result. The per example strong convexity assumed here is not common in the literature at all (typically it is average loss strong convexity), so this result adds little value to the paper. 

The reviewers also highlighted multiple weaknesses in the evaluation and method, such as scalability in practice due to the projection step and the use of non-standard models in evaluation. These should be properly acknowledged in the revision.
confidence: Certain
Decision: Accept