Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors study a new and interesting problem, i.e., they consider the heterogeneity of both the labels and tasks in personalized federated learning, which is very different from the assumptions in previous works, i.e., a same set of labels and a same task in different clients. The studied problem, termed as the Agnostic Personalized Federated Learning (APFL) problem, is well motivated, and the associated challenges are well identified.

Then the authors then propose a novel solution, i.e., Factorized-FL, for the studied problem. Specifically, Factorized-FL factorizes model parameters to reduce the parameter in order to alleviate the knowledge collapse, and utilizes them to measure the task-level similarity for matching relevant clients. Empirical studies on public datasets also show the effectiveness and efficiency of Factorized-FL.

strengths_and_weaknesses: Originality:
It is an interesting and novel work though the technique is relatively simple.

Quality: 
The main idea is simple, i.e., factorization of the parameter into a sharing part and a non-sharing part.

Clarity: 
The paper is well presented with good illustrations and explanations.

Significance:
It is a nice work which addresses an important problem, i.e., heterogeneity in labels and tasks, and achieves significant improvement over the SOTA methods.

questions: 1 Is there some simple approach to address the label heterogeneity challenge, e.g., label alignment before training?

2 In federated learning, privacy protection is an important issue. In particular, in the similarity matching step, the calculation of similarity requires V of other clients, which may disclose the privacy of the clients. However, the authors do not discuss about the privacy issue. Is there some theoretical guarantee about the privacy protection?

limitations: The authors are suggested to include more discussions or theoretical results on privacy protection.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new Agnostic Personalized Federated Learning (APFL) problem and study the personalized federated learning method with parameter factorization and similarity matching.  Specifically, the authors focus on label heterogenous and domain heterogeneous problems.  All in all, the proposed method shows good performance compared to other personalized federated learning baselines.
strengths_and_weaknesses: Some strengths:

1. The paper tackles an interesting problem with personalized federated learning. The paper has very extensive experiments with great analyses. The contributions are good.
2. According to Table 1 on page 7, Factorized-FL (both versions) shows good results across datasets and scenarios (domain and label heterogeneous) 
3. It’s also good that the authors provided extensive experiments on the Appendix as well as source code for reproducibility.


Some of my concerns/suggestions for improvements: 

1. Could you please elaborate more on why rank-1 vectors with sparse matrix? What if we consider other values of rank gamma != 1 with local bias (Figure 3 left)? I did not see (or may miss) the ablation study of different rank values, and perhaps discussions of trade-offs for rank > 1. I would love to see more analyses rather than only Figure 7 (h). (Same for Figure 7 (i))
2. In line 46-50, could you please elaborate more on why permuting the labels and learning on different datasets make gradients more diverge? 
3. In Figure 2 (c), if we run for more epochs, what are the performance? Will it still look ‘consistent’ as in the current version? 
4. Why Factorized-FL alpha and Factorized-FL beta performs ‘inconsistently’ compared to each other in Table 1 page 7 w.r.t both Label-heterogeneous and Domain-heterogeneous?
5. [Minor] Figure 7 (a)-(e) are hard to see.


Overall, the paper is good in my opinion. 

questions: Some of my concerns/suggestions for improvements: 

1. Could you please elaborate more on why rank-1 vectors with sparse matrix? What if we consider other values of rank gamma != 1 with local bias (Figure 3 left)? I did not see (or may miss) the ablation study of different rank values, and perhaps discussions of trade-offs for rank > 1. I would love to see more analyses rather than only Figure 7 (h). (Same for Figure 7 (i))
2. In line 46-50, could you please elaborate more on why permuting the labels and learning on different datasets make gradients more diverge? 
3. In Figure 2 (c), if we run for more epochs, what are the performance? Will it still look ‘consistent’ as in the current version? 
4. Why Factorized-FL alpha and Factorized-FL beta performs ‘inconsistently’ compared to each other in Table 1 page 7 w.r.t both Label-heterogeneous and Domain-heterogeneous?
5. [Minor] Figure 7 (a)-(e) are hard to see.


limitations: The authors did have clear discussions about limitations and potential negative societal impact in Section A in the Appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a personalized federated learning method based on parameter factorization and similarity matching. The authors factorize the model parameters into a pair of vectors to condense common and personalized knowledge, which are used for guiding the model update aggregation. Extensive experiments show that the proposed method can better handle both label and domain heterogeneity of data in federated learning than many existing methods.
strengths_and_weaknesses: Strengths:
1. This paper studies a setting with stronger data heterogeneity.
2. The proposed method is interesting.
3. The experimental evaluation is solid.
4. This paper is well written.

Weakness:
I only have one concern on the experimental setting. The authors group the 100 classes to simulate the domain heterogeneous scenario, but this setting is quite similar to the standard extreme label heterogeneous scenario where each client only keeps one class of samples. Thus, the boundary between label and domain heterogeneous scenarios are somewhat vague. I think a simplified scenario where different clients keep different domains of data but their label space is similar (a good example is review sentiment analysis), is also interesting.
questions: N/A
limitations: The authors have adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper considers two new heterogeneous FL settings, (a) hetero-label FL where the clients have different labeling orders, i.e., the same label space but the ground-truth ids are different permutations across clients, and (b) hetero-domain FL where the clients have disjoint datasets in both instance and label class. To deal with these two challenging scenarios, the authors propose to factorize the model parameters into the product of two vectors plus a sparse bias term, in which the first vector is personalized for a specific client and the second one is shared with all participants. Experiments on CIFAR-10/100 and SVHN datasets with various partition manners show promising results with the proposed method. The studied problem and proposed method are interesting, and the paper has a clear organization. However, I have several concerns about the practicability of the hetero-label FL setting, the writing quality, and the soundness of the paper.

strengths_and_weaknesses: Strengths:
1. The proposed hetero-domain FL setting is novel and practical, which has good potential to extend the application and research scope of FL.
2. The paper has a clear organization and the proposed factorization method is easy to follow.
3. The authors propose to capture client-general and client-specific knowledge via a global-shared base vector and local-personalized coefficient vectors, respectively, which is intuitive and seems to be simple yet effective.

Weaknesses:
1. The practicability of the hetero-label FL setting is questionable. I agree that clients may have different label spaces as different labeling schemes can be adopted. However, the label permutation seems a bit strange and tailored for the simulation, as the clients in realistic applications can easily reorder the labels to make their label orders consistent with negligible computation and communication costs (at the beginning of the FL courses and communicating only once).
Further, if reordering information is forbidden to be exchanged, the users can also leverage privacy-preserving entity resolution techniques to align the label IDs, which are well-studied in vertical FL literature such as [1].

2. The proposed factorization method is simple, while the authors have not given more in-depth analysis and discussions on such topics as (1) The convergence of the proposed method since the converged communication round is also important in FL; (2) The expansibility of the proposed method in terms of the model structures. The authors focus on the fully-connected and convolutional layers. Can the proposed method be applied to other popular and practical NN models containing BN layers, residual structures, attention structures, etc., (3) The space-time complexity analysis with the proposed factorization method. The authors claimed that the proposed method improved FL efficiency multiple times. However, a more formal time-space complexity analysis is lacking in terms of FL training, inference, and communication processes.

3. The experimental datasets should not be applied with the label permutation when examining the performance in the domain-heterogeneous FL setting (see line 265 and paragraph 280). The permutation mixed the effect of label heterogeneity, making the analysis in paragraph 280 of domain heterogeneity confounded. Besides, the number of experimental clients is a bit small (20 clients usually correspond to cross-silo FL). How does the proposed method perform on larger client scales and cross-device FL scenarios?

4. The writing has a lot of room for improvement in terms of preciseness, typos, and grammatical mistakes. For example,
- line 23, "the same set of labels that are in the same order" The elements in a set are *un-ordered*.
- line 43, the numbering of the figures and the order in which the figures appear do not match: Figure 1 appeared after Figure 2
- line 44, the term "instance-wise manner" is un-clear
- line 36 ~ 50. the example is a bit hard to read and it is unclear whether the authors make the classifier layers personalized to each client in the example. Without the classifier layer personalization, the gradients' divergence is obvious and comparison is trivial. 
- line 90, downloads -> download
- line 95, use -> using
- line 97, introduces -> introduce
- line 115, initialize -> initializes
- line 127, utilized -> utilize
- line 129, wher -> where
- line 139 and line 149. there are two notations for the client datasets, P_k and D_k. The authors used D_k in equation (6) and how about P_k.
- line 151, the last equation only expresses that there is no instance-wise overlap, while not mentioning the class-wise overlap.
- line 169, uses -> use
- line 177, shows -> show
- equation 1 and equation 2 lack periods
- line 211, another term "task-heterogeneous FL", compared to the term "label-hetero FL", it is better to unify them
- line 213, demonstrated -> demonstrate
- line 214, achieves -> achieve
- line 250, Models -> Baselines,  This section lacks an introduction to the adopted models
- line 257, missing citations for CIFAR-10 and SVHN
- line 259, which labels -> whose labels
- line 331, "As shown, " incomplete sentence
- in appendix Figure 4, there is a reference error: "Table ??"

Reference

[1] Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption

questions: See questions mentioned in the above weaknesses. 
limitations: IMO, there is no potential negative social impact.
review_title: null

Meta-review: metareview: In this paper, the authors study an interesting problem where clients are heterogeneous in both labels and learning tasks/domains. This problem is different from most of the existing pFL settings, and has great potential to broaden the application of FL. To handle this case, the authors propose a novel method based on parameter factorization and similarity matching. Experimental results on both label and domain heterogeneous cases are promising. To this end, I recommend accepting this submission.

However, I do have some concerns regarding the experimental setting of the heterogeneous domain (as pointed out by reviewer BK7j and dsaH). Without proper dataset simulation, it makes the experimental results less convincing.  

This submission can be further improved from all the discussions between reviewers and the authors. Hope they find the discussion useful and make this submission a better one

confidence: Less certain
Decision: Accept