Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new learning framework to tackle the label distribution skew challenge in federated learning. Specifically, the paper introduces a novel feature representation alignment technique and a classifier combination technique to exploit both shared representation and inter-client classifier collaboration. By using a local regularization term related to the global feature centroids of each class, the framework mitigates the drift of local features during the learning process. By introducing a linear combination of classifiers with optimized coefficients during training, the framework allows similar clients to share more information while avoiding negative transfer from more distinct clients.
strength_and_weaknesses: Strengths: The proposed method is well-motivated. The use of global feature centroids in local training is straightforward and practical, and the idea of forming classifiers combination coefficient selection as a quadratic programming problem is elegant. Evaluation and ablation studies seem convincing.

Weaknesses: The proposed framework tackles the specific label distribution skew challenge in federated learning. It would be nice if the authors could discuss whether the method generalizes to other challenges caused by heterogeneous data distributions in federated learning, for example, when the number of classes is different (if so, whether the method needs any adaptations; if not, why the method has certain limited applications).
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The work is original to my knowledge.
summary_of_the_review: This paper introduces global feature centroids in feature regularization and classifier combination weights selection through quadratic programming to improve personalized federated learning. The proposed method is backed by good intuitive interpretation, theoretical analyses, and convincing evaluation results. The ideas introduced in this paper could be inspiring to the relevant community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to deal with the data heterogeneity in federated learning by local-global feature alignment and weighted combination of local classifiers. The whole model is decomposed into two parts, a feature extractor $f$ and a linear classifier $g$ built on $f$. The feature alignment is achieved by extracting feature centroid for each label class. The parameters of each local classifier $g_i$ is combined with other classifiers by optimized weights so as to bundle similar local clients. 
strength_and_weaknesses: **Strength**
1. The proposed framework is novel. The decomposition of feature learning and linear classification helps customize local clients.
2. The performance is theoretically analyzed based on a simplified model and numerically validated by experiments on several datasets.

**Weakness**
1. The heterogeneity of data considered in this paper is a specialized one. Generally, the heterogeneous local data distributions will be different in both total number of samples and number of feasible label classes. Current setting assumes each client has the same size of data and observes the full set of label classes. It is interesting to see the performance of the proposed method when the local data distributions are heterogeneous enough.
2. In Eq. (3), the authors assume there only exists one centroid for each label class. This assumption needs to be better explained. It is likely that samples in one label class may come from two different centroids. Simply averaging all the features as one centroid may not always be true.
3. In Eq. (4), the authors mention $\alpha$ is optimized to reduce the local testing loss. It is not clear whether the local test data is different from the test data used to show Table 1.



  
clarity,_quality,_novelty_and_reproducibility: The paper proposes a novel idea and the presentation is good. There may exist some flaws mentioned above.
summary_of_the_review: The paper is overall well-written but there are some critical issues that need to be addressed. I'm happy to increase my evaluation if my concerns (see weakness) are addressed.

----------- Post rebuttal ----------

All of my concerns are well-addressed. I increased my evaluation from 5 to 8.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work solves personalized federated learning (PFL) setting by developing feature alignment mechanism and classifier collaboration. Concretely, this feature alignment minimizes the distance between local feature representations and global centroids. And classifier collaboration utilizes linear combination of multiple local classifiers to customize new classifier. The current work conducts experiments on four datasets to verify the effectiveness of proposed method.
strength_and_weaknesses: Strength:

1. The improvement of experimental results is significant.

Weaknesses:

1. The description of some details in the proposed method is not clear.

2. Need to make comparisons with some recent PFL works to verify the effectiveness.

clarity,_quality,_novelty_and_reproducibility: The writing part should be improved, and the novelty of algorithm is limited.
summary_of_the_review: This work solves personalized federated learning (PFL) setting by developing feature alignment mechanism and classifier collaboration. Concretely, this feature alignment minimizes the distance between local feature representations and global centroids. And classifier collaboration utilizes linear combination of multiple local classifiers to customize new classifier. The current work conducts experiments on four datasets to verify the effectiveness of proposed method.

However, I have several concerns on this work.

1. Eq. (1) is the optimization goal of traditional federated learning (FL) instead of that of PFL task.

2. The feature alignment regularization needs to access the global centroids which are obtained by aggregating local centroids. Under this condition, the data distribution information of each client has been leaked out, which violates the requirement of privacy protection in PFL task.

3. In classifier collaboration, why the simple linear combination of local classifiers can find these clients with the similar distributions? It would be better to analyze the combination coefficients to support this point.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose FedPAC for personalized federated learning, which aims to learn a better representation by leveraging global semantic knowledge. Comprehensive experimental results multiple benchmarks demonstrate the effectiveness of the proposed framework.
strength_and_weaknesses: Strength

The structure of the proposed FedPAC is reasonable and easy to follow.
The studied problem (personalized federated learning) is interesting and different applications may benefit from it.
the experiment results are extensive and impressive
The authors also give a theoretical analysis.

Weaknesses

The description of motivation could be improved.
The paper could benefit from describing personalized federated learning in more detail.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written and the presented results are satisfactory. The evaluation is carried out on public datasets, which can facilitate future comparisons with novel methods. An ablation study is also given to evaluate the contribution of the different losses proposed by the paper. In general, the proposed DREAM framework is not difficult to reproduce.
summary_of_the_review: I am generally optimistic about this submission. Considering the well-motivated problem, novel framework and impressive experimental results, I have no negative comments on the quality of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Four experts reviewed the paper and provided ratings of 8, 8, 8, and 5, respectively. The reviewer who remained negative post-rebuttal was mainly concerned about data privacy leakage, but the other reviewers were satisfied with the rebuttal. Therefore, AC decided to side with the majority of reviewers as there was no further response from the negative reviewer. Hence, the decision is to recommend the paper for acceptance. The authors are encouraged to make the necessary changes to the paper to the best of their ability following the reviewers' comments. 
justification_for_why_not_higher_score: A reviewer remained concerned about the data privacy leakage. 
justification_for_why_not_lower_score: Three reviewers rated it 8. They liked the paper's intuition, theoretical analyses, and experiments.
Decision: Accept: notable-top-5%