Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The main contribution of the work are as follow: 
1. introduce the problem of Agnostic Personalized Federated Learning (APFL), and discuss its two possible issues (Label- and Domain-Heterogeneity)
2. propose the method called Similarity Matching and Kernel Factorization (SimFed) to tackle the problem. 
3. Validate the method in both label- and domain-heterogeneous scenarios, and show its superiority.  
main_review: Strengths: 
Well motivated challenges to tackle: label heterogeneity and domain heterogeneity. Then the authors propose similarity matching and kernel factorization to address the issues and show compelling results in experiments. 

Weakness: 
1. In table 2, it shows that Similarity Matching + Factorization did not outperform Stand-Alone for the "Fruits & Vege" dataset. It would benefit if authors could clarify why, and share some intuitive explanation why or in what scenarios that the proposed algorithms underperform comparing to the "stand alone" approach. 
2. The paper would benefit with some explanation why each baseline models & architecture were chosen here for comparison. 
3. It was not clear why SimFed reduces information loss when reflecting the heterogeneous knowledge. 
4. Kernel Factorization outperform SimFam in most datasets, excepts some. It would benefit if the paper could explain why certain dataset behaves that. 

Minor typo: 
1. In page 2, it says "We exhaustively validate our method on both singl[e] domain datastes (Permuted IID/Non-IID) and multi domain datasets (CIFAR-100 20 superclasses) and show our method outperforms the current state-of-the-art approaches on all experiments." It should be "single" rather than "singl". 

summary_of_the_review: marginally above the acceptance threshold, given that the weakness shared above. Overall it was well motivated problem, and the authors proposed approaches and show compelling results. As pointed out in the above for its weakness, the paper would be strong by providing these insights and intuitions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates the FL under the label and the domain heterogeneity.
Specifically, it considered two settings where (i) the labels may not be precisely aligned (ii) the data domain may not be the same. To tackle these two challenges, it proposed the SIMFED framework which first calculates the similarity between the embedding of different local models and then conducts a kernel factorization to solve the problem of heterogeneous data domains.
main_review: I have the following major concerns.

- The comparison baselines for Label Heterogeneity do not seem informative. By design, many of these baseline methods would implicitly require the labels to be correctly aligned. A more reasonable way is to compare the proposed framework to those methods that take label mismatch into account. For example, you may simply integrate permutation-robust techniques into FedAvg to deal with label heterogeneity cases. I would like to see more comparisons based on such kinds of baselines.

- It is unclear why we should use the cosine similarity. Each local model may have different preferences of scaling resulting given different domains of the data source. Also, what exactly/mathematically does the keyword ’semantic’ mean?
summary_of_the_review: Overall, I found the paper is a heuristic work on personalized federated learning with little justification and little motivation. The scope of its application is unclear. The empirical results do not seem to be sufficient to support the claimed contributions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors propose an algorithm for clients to learn from each other even under the setting of label heterogeneity and domain heterogeneity. The basic idea is they leverage metrics to measure the similarity between the representative knowledge and corporate clients accordingly, which can maintain the generally shared information and reduce information loss at the same time. They conduct appropriate experiments to test the algorithm and provide a comparison with current the-state-of-the-art fl methods. The main contribution is to propose an approach to support clients to benefit each other across different types of labels and domains. They leverage the kernel factorization approach on FC and convolutional layers and update them separately on the server-side. They provide extensive experiment results to support their claims.
main_review: Strengths:
+ The research problem that benefits clients by sharing some common knowledge with a well-designed mechanism is an interesting topic
+ To decompose parameters to lower level rank for local models, to my best knowledge, is a novel idea to explore.
+ The experiment results are sufficient enough to support their claims

Weakness:
- There are some research works discussing how to use general knowledge across clients by dividing parameters into different parts, like continual learning in FL, where each client has several tasks to conduct [1]. Also, the basic idea to group users based on metric similarity has been explored in FL as well [2,3];
- I am a little concerned about the privacy guarantees and extra computational requirements on the server-side. The extracted knowledge from the clients is leveraged by reflecting the difference between the target local model and the other clients. One of the benefits of using FL is protecting users’ privacy. Though clients are not sharing data in this work, we need to pay attention to how to leverage extracted knowledge. Also, the kernel factorization of FC layers and convolutional layers happens on the server-side. I am not sure how much extra computation it would take and whether it would cause a delay in the iterative FL update process.
- More discussion about the effectiveness of similarity matching and kernel factorization is expected. Furthermore, in most FL work, convergence guarantee is also another point that is expected to discuss besides the experiments.

[1] Yoon, Jaehong, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. "Federated continual learning with weighted inter-client transfer." In International Conference on Machine Learning, pp. 12073-12086. PMLR, 2021.

[2] Duan, Moming, et al. "FedGroup: Efficient Federated Learning via Decomposed Similarity-Based Clustering."

[3] Jeong, Wonyong, et al. "Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning." arXiv preprint arXiv:2006.12097 (2020).

summary_of_the_review: The authors do provide a fine solution to solving a real problem of generalizing knowledge across clients by kernel factorization. Extensive experiments are provided as well. However, if we consider each part of the techniques, like model parameter kernel factorization, grouping users based on similarity, I may expect more novel contributions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a personalized federated learning algorithm to deal with label heterogeneity and domain heterogeneity. Specifically, the paper focuses on the cases where clients performs classification tasks while the label set of data samples are vary among clients. Furthermore, the data samples may be distributed among clients from different domains. 
main_review: The problem studied in this paper as well as presented experimental results are interesting. However, the paper has some weaknesses:

1- It seems that the paper assumes that models, are neural networks (either with dense layer of convolutional layers). However, this is not mentioned in sections 1 and 3. Furthermore, this paper studies only classification problems. However, authors do not make this clear in the paper especially in section 3. Therefore, the problem setting and assumptions are not clearly defined in the paper. The authors are suggested to revise section 3 to address this issue.

2- The paper needs more rounds of proof-reading and there are some grammatical errors particularly in section 4.1 which makes understating this paper difficult. I think the paper is not well-written and the presentation of the paper in this sense should be improved. For example, in section 4.1, the sentences "Our underlying assumption is that we can discriminate which local models are helpful or harmful each other if we can differentiate which local data distributions are semantically relevant or irrelevant." and "we can measure similarity between the local domains in the embedding space, which eventually enables us to determine which models are beneficial each other accordingly." are not clear that what they want to express. In fact, the authors do not make it clear that what it means by stating the models benefit each other. I think the authors should specify that in what sense the models can be harmful or helpful to each other.

3- There are some personalized federated learning algorithms as well as agnostic federated learning algorithm. I am curious that what happens if we can simply combine these algorithms together to obtain some agnostic personalized federated learning algorithms. It would be great if the authors can compare the performance of their algorithm with some algorithms which are trivial combination of agnostic federated learning and personalized federated learning. This can better shows the effectiveness of the proposed algorithm dealing with label heterogeneity.
summary_of_the_review: Overall, I think the paper studies an interesting problem. However, there some issues in the paper that need to be addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies two challenges of personalized federated learning: (1) Label Heterogeneity where label schemes are not synchronized in local clients and (2) Domain Heterogeneity where the datasets owned by the clients can be semantically dissimilar. The authors propose a method called Similarity Matching and Kernel Factorization which measures semantic similarity/dissimilarity between locally learned knowledge and aggregates the relevant ones. Furthermore, the method factorizes the model parameters into two basis vectors and sparse masks to capture representations of the heterogeneous knowledge.
main_review: Pros: 
(1)	The two scenarios in federated learning studied in this paper including label heterogeneity and domain heterogeneity are interesting and practical. 
(2)	The proposed method views the individual local models as respective encoders that can transform their local data distribution to embedding vectors shared among clients, instead of adding any secondary models, which is simple but effective.
Cons:
(1)	In Section 4.2, authors assume that if the dimensionality of parameter space to compute is significantly reduced as well as knowledge is separately captured, then the loss of essential information of model aggregation in domain heterogeneity scenario can be alleviated, which is not well justified.
(2)	There is no reason why the basis vector parameters u can capture the input-related knowledge and v can learn the output-related knowledge.
(3)	There is no detailed justification or no reference to key formulas in the paper. The Eq. (8) for updating factorized parameters is mistaken.
(4)	Some experimental results in Table 1, Table 3 and Table 4 are missing, and the setting of similarity threshold is not presented, which harms the reproducibility of the proposed method. The variant method “w/o Factor” in Table 4 is confusing.
(5)	Given the heterogeneity of different local datasets, the projection these datasets into a latent intermediate space for all clients can not well ensured. The authors did not give sufficient rational analysis for this assumption. This operation may destroy the privacy leak of local data.
The studied problem in this paper is interesting and practical, but the technical idea and technical contents are not well justified, and the experimental results are not well organized and analyzed.

====
Comments after rebuttal:
First of all, the authors make good efforts during the rebuttal period and clarify my concerns on privacy issue. There are still several issues not well addressed.
1. I just checked the author's response and the comments from other reviewer. From my own experience in personalized federated learning, existing solutions indeed can handle different label spaces or non-i.i.d. data across clients. FedBN (ICLR2021) can handle the feature heterogeneity. The authors do not well position their own work and the main technical idea is not well analyzed yet. I admire the addtional results and more ablation study.
2. The authors still do not well revise Eq. (8), the subscript j for the numerator is not well referred from the context, why not merge with the summation also not discusssed.
3. The kernel factorization is not well analyzed yet, mostly are simply supported by improved results.
4. Overall, the main technical idea is not well analyzed and manuscript is not so clearly presented yet. I would like to keep my scorings.
summary_of_the_review: This paper studies the novel and interesting scenarios in federated learning, i.e. label heterogeneity and domain heterogeneity, and the experimental results show that the proposed method outperforms other federated learning methods on single- and multi-domain datasets. The main logical structure of the paper is relatively clear. However, there are also some shortcomings that need to be improved, e.g. the proposed assumption is not well justified and the experimental setting should be detailed. Overall, this manuscript is not mature enough.
review_title: null

Meta-review: The paper studies two aspects of personalized federated learning: (1) Clients having their own labeling scheme. (2) Domain heterogeneity across clients. They propose a way to collaborate across clients by similarity matching. The key novelty is to measure similarity of client pairs, based on on how much their representation layer agrees (measured with cosine similarity). A second novelty is a low-rank factorization of model weights. Empirical evaluations show wins on MNIST, CIFAR10, 100.  

Reviewers had various grave concerns. On the method side, they were concerned that thee is not enough theoretical insight and analysis of the proposed approach, esp. the kernel factorization and its effect.  On the empirical side, they were concerned that comparisons were not made with most recent baselines. There was a large number of PFL approaches published in 2021. e.g. FedBN.  Among these, its worth noting pFedHN (ICML2021) which actually discussed the case of heterogeneous (permuted) labels (their Sec 3.3).

In a discussion, reviewers appreciated the responses by the authors, the additional experiments and ablation studies. Unfortunately however, they found that the paper is not ready for publication in ICLR.
Decision: Reject