Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper mainly focuses on a practical scenario of FL where the labels might be marked by different criteria over different centers/nodes. A new learning strategy is proposed for dealing with the mismatches of the labels, and it can be deployed with existing FL methods. Theoretical analysis shows that the proposed FedMT is able to find the global optimal solution for this class of problems at a linear convergence rate. Multiple numerical results further verify the intuition of this method with the experiments on real datasets.
strength_and_weaknesses: The strength of this work is mainly about the consideration of a new setting in the FL system. The mismatch among the center do exist in practice. To the best of my knowledge, this would be the first work addressing this issue. The proposed algorithm seems simple and easily implementable. The numerical results are encouraging. 

The main weaknesses of this work are as follows:

1) it is quite confusing about the projection matrices. It seems that this idea is borrowed from the other (centralized one). There are many issues missing, e.g., how to choose these matrices so that they are invertible? it seems that they are given as prior. Then, why do claim the agonistic? 

2) The convergence results are built upon the NTK regime. Why is this regime considered? Convergence on the first-order stationarity for nonconvex loss function is widely used and applicable for general problem beyond neural nets.

3) It is claimed that even SGD used in this work can achieve the linear convergence rate and a step size of independent on iteration is used. Then, how about the gradient descent case? How does stochastic gradient estimate error play into the convergence result?


clarity,_quality,_novelty_and_reproducibility: Clarity: as mentioned above, there are several issues, e.g., design of T,Q, convergence results, which are clearly discussed.

Quality: the theoretical part may contain some errors. SGD generally cannot achieve the linear convergence rate, even in the strongly convex case. The number of inner loop $t$ times the step size is appeared in the convergence result, meaning that if the server and the nodes do not communicate frequently, then, the local nodes are required to choose small step sizes (which is inversely proportional to $t$). Why does not use a large step size? This claim is not strong. 

Novelty: I believe this setting is new. The error correction idea is actually borrowed from the centralized one directly. If the major differences between these settings can be further addressed, the novelty of the designed method should be good.

Reproducibility: the code is available at an anonymous link. 
summary_of_the_review: In summary, the work indeed brings the mismatch issue in the FL setting, but a thorough discussion on FedMT should be carefully studied, including projection matrix design, convergence analysis, communication efficiency, etc.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors address the problem of federated learning in the scenario where clients have differently defined label sets (assuming a super set of labels and a subset with and without partial overlap), but without requiring cross-client data relabeling. Further they consider also the setting where only a small portion of the data (in the specialized server) is labeled and labels may be noisy, They also provide theoretical results concerning the convergence of the proposed approach.
strength_and_weaknesses: The proposed approach is well motivated, justified from a practical perspective and supported by theoretical results. The experiments demonstrate both the performance of the model relative to competing approaches and support the theoretical findings. The main weaknesses are: i) the need for known and fixed Q and T matrices, ii) the assumption that only the specialized server suffers from label noise, iii) that though the proposed approach is proposed as a federated learning approach, the fact that only two sets of labels are considered (one containing the other) reduces the federated aspect to the model to simply applying FedAvg with the corrupted label learning of (Van Rooyen & Williamson, 2017; Patrini et al., 2017), which can be seen by comparing Algorithms 1 and 2. Note though that the latter could be perceived as a strength of the proposed approach except that it clearly separates the members of the federation as i) those with the larger set of labels which may be affected by noise, and ii) those with a smaller but common set of labels that are not only cheaper to obtain but are noise free. These two assumptions greatly limit the applicability of the proposed approach in general federated learning settings.

The authors make the assumption that there are only two label spaces, however, such an assumption lacks a practical justification, more so considering the original motivation of the approach (from the paper: "standards for disease diagnosis may be different across clinical centers due to varying levels of expertise or technology available at different sites"). Though, the authors claim that extension is easy, it is not clear why being easy, is not explored.

The structure of the proposed framework raises several questions that are not addressed in the paper:
- why there is a need for a server that both handles the aggregation but also contributes data. This is not standard in FL, arguably unnecessary, and may result in privacy issues or at the very least power imbalance issues (the specialized server has access to the model, and updates from all the clients while at the same time having a local model and access to data).
- why only assume that the specialized server has noisy labels?
- from the text it is not clear under which assumptions does Theorem 3.1 holds and how those relate to the practical setting considered by the authors.

For a benchmark experiment, it is justifiable that T and Q are not only known but easy to obtain, however, in practice, estimating T is in general difficult and Q is complicated for non-nested cases.

The paper specifies that the PD data from Qin et al. (2020) is synthetic, however, other places in the text and the supplementary material suggest that the data is real. Can the authors please clarify? The reasoning for the question is to better understand how realistic are the assumptions about Q and T in relation to whether the data, endpoints, noise or sub-classes (with overlap) were artificially crafted.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow, the proposed approach is clearly motivated and supported by theoretical findings. Novelty is somewhat limited considering the straightforward extension of the work in (Van Rooyen & Williamson, 2017; Patrini et al., 2017) to the federated learning setting, however, it is worth noting that the authors leverage results from NTK to better understand the convergence of the proposed approach, even considering the effects of Q and T on convergence.
summary_of_the_review: The proposed approach constitutes a first step in the development of federated learning systems with nested or partially overlapping endpoints, while allowing for clients/servers with small sample size and label noise. However, the assumptions (known and fixed Q and T, one common subset of labels for all clients, noise labels only in the server, etc.) and experiments underscore the limitations of the proposed approach. Specifically, it narrows down the applicability of the proposed approach to very specific and arguably rare practical situations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose to tackle the potential mixed-type label problem in federated learning. Specifically, two types of projections are introduced to both server and clients to achieve the alignment, which can be also generalized to noisy label scenario. The authors also provide theoretical convergence analysis. In the experiments, the authors conduct evaluation on CIFAR-100 and sEMG datasets, the proposed projections can significantly improve the performance compared to other SOTA federated learning baselines.
strength_and_weaknesses: Pros:

The paper is well-written and well-motivated. The discussed topic is challenging and important to the federated learning in real scenarios. The proposed projections are easy to follow and seem natural for the mixed-type labels. The authors provide theoretical analysis of the convergence of proposed algorithm. The experiments are extensive. The evaluation is conducted on various datasets and the proposed algorithm is compared to various SOTA baselines. The results are promising and convincing.


Cons:

My major concern lies in the advantage of two proposed projections. The authors introduce two projections align the prediction of different types of labels to the true label space, which are applied to probability and label respectively. Although the theorical analysis focuses on the label projection and shows better convergence, the results in Table 1 and Table 2 show the superiority of probability projections over label projections. It is difficult to see the necessity of proposed label projections.

clarity,_quality,_novelty_and_reproducibility: This paper targets an interesting scenario in federated learning where each center has different labeling criterion, and the proposed algorithm is also simple and efficient, which could benefit the federated learning community. In terms of reproducibility, the authors provide clear description of experiment setup and the details are also included in the supplemental material.
summary_of_the_review: Overall, I think this paper is interesting and solid. It would be better if the authors can provide more discussion of two types of projections and demonstrate the scenario where probability/label projection shows superiority.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose a method that implements the federated learning(FL) with mixed-type labels. In detail, the authors propose a theory-guided and model-agnostic approach that can make use of the underlying correspondence between different label spaces and can be easily combined with various FL methods. The proposed method achieves better results than the baseline methods.
strength_and_weaknesses: Strength:
1. This paper is the first work to investigate FL’s theory on neural network classification under mixed-type labeled data.

2. This paper is easy to read.

3. The authors propose a theory-guided and model-agnostic approach that can use the underlying correspondence between those label spaces and can be combined with various FL methods.

4. The proposed method achieved relatively good performance than the baseline methods.

Weaknesses:
1. The authors implement the experiments with the CIFAR100 dataset and ResNet18. I am curious about the experiments with large datasets, such as ImageNet, or lightweight architectures, like MobileNetV2.

2. The authors propose two kinds of projections for servers and clients, as in Eq. (3) and Eq. (4). I am curious about these two projections' effects on the proposed algorithm.

clarity,_quality,_novelty_and_reproducibility: This paper is easy to read, and the novelty is relatively good.
summary_of_the_review: In this paper, the authors propose a method that implements the federated learning(FL) with mixed-type labels. In detail, the authors propose a theory-guided and model-agnostic approach that can make use of the underlying correspondence between different label spaces and can be easily combined with various FL methods. The proposed method achieves better results than the baseline methods. The novelty is relatively good, and the proposed method achieved a relatively good performance than the baseline methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims to tackle a novel practical FL setting with mixed-type label classification (e.g., disease diagnosis where clients use different label spaces). As the first work, the authors propose a versatile method, FedMT, along hew insights into method design inspired by theory. Their experiments validate the key factors and show better performance than alternative solutions. 

Main strengths are:
- Novel setting, definitely bringing some new ideas to the FL field
- Simple algorithm (a variant of FedAVG) based on projections, which are backed by NTK theory-derived proofs
- Consistent gains over experiments
- Code is available and details are included in the appendix.

Main critiques are: 
- The setting might be a bit artificial
- Limited novelty w.r.t. (Rooyen & Williamson, 2017) 
- Clarity issue of two projections; lots of typos in the original submission
- Theory is very limited (due to NTK-based), and seems detached from practice
- Improvements are often marginal
justification_for_why_not_higher_score: The paper has certain promise in its settings and experimental results. But one consensus is that the theoretical contribution is thin in this paper. It should be mainly appreciated as a simple, empirical algorithm in an unexplored setting. More outstanding issues include the confusing description and non-trivial estimation of matrices Q and T, as well as the lack of discussion about the combination with more advanced FL algorithms. The paper's clarity, as well as self-consistency, also needs more revision work. Therefore, AC recommends the rejection.
Decision: Reject