Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies federated learning where the local data on each client are non-iid in the sense that they may encounter openset noisy labels. To solve the problem, they extend the so-called peer loss which is studied in centralized label noise learning to the federated learning scenario and call the proposed method FedPeer. They also test the FedPeer method on both benchmark and real-world datasets.
strength_and_weaknesses: Strength

The paper considers federated learning with openset noisy labels, which may be practically useful. The proposed method is tested on both benchmark and real-world datasets and is shown to outperform baseline methods.

Weaknesses

The first concern is about novelty. As peer loss is well studied in centralized label noise learning by Liu & Guo 2020 and Cheng et al., 2020, the main contribution of the current paper is to extend it to the federated learning setting which seems quite straightforward.

Another concern is about the convergence of the proposed method. As the proposed method contains multiple steps, e.g., initialization with $T_{DP}$ generation, label communication, loss evaluation on noisy data, and peer gradient updates, the rate of openset noise and the selection of $\epsilon$ may inevitably affect training, how can we guarantee the convergence of proposed method?

The standard setup of federated learning is to train a model collectively from multiple parties without sharing each party's local data. The label communication step in FedPeer seems necessary to FedPeer but violates the standard federated setting. Could the authors provide some real-world examples of federated learning with openset noisy labels and justify label communication with the label privacy guarantee is enough?
clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well-written and very easy to follow. 

Quality:

The paper considers federated learning with openset noisy labels and extends peer loss to the federated setting.

Novelty:

Given the existing work on peer loss in centralized label noise learning by Liu & Guo 2020 and Cheng et al., 2020, the novelty of the current paper is not strong.

Reproducibility: good
summary_of_the_review: The paper is generally interesting to me, but the novelty part is a concern and needs to be discussed carefully, e.g., what are the difficulties and challenges in the federated setting? Analysis of the convergence of the algorithm and more real-world examples are also needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript studies an interesting and non-trivial problem, which is how to learn with open-set noisy labels under federated settings. The authors first give a very clear and easy-to-understand formulation of the federated open-set label noise problem, which is very useful. They then proposed a federated label-noise learning method by leveraging contrastive labels, and a Differential Privacy (DP)-preserving scheme to collect noisy labels without violating federated learning settings.
strength_and_weaknesses: Pros:
1. The idea of using a transition matrix with a DP guarantee to generate DP label is interesting, I'm not sure if it is new (I'm assuming it is).
2. The research problem is clearly defined and well-supported by a concrete example, and the solution is strongly motivated.

Cons:
1. The authors should discuss how and why the DP label can still form a complete noisy label space. Based on my understanding, there is no guarantee that the DP label after flipping can still in fact form a complete noisy label space.
2. If the DP label transition matrix is generated by the server, wouldn't the server be able to invert the uploaded DP label with their respective T?
3. Authors' example on page 4 seems not correct, if a local T is estimated based on a label space without class 3, the third row of T should be all zeros. Specifically, based on commonly used anchor-point noisy posterior estimation [1,2], since there is no anchor point for class 3, the third row of estimated T should not exist (or all zeros).
4. The compared baseline methods should include some more recent methods and some state-of-the-art methods that also don't leverage the Transition Matrix for a fair comparison, such as Divide-Mix [3].

[1] Liu, T., & Tao, D. (2015). Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3), 447-461.

[2] Xia, X., Liu, T., Han, B., Wang, N., Gong, M., Liu, H., ... & Sugiyama, M. (2020). Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33, 7597-7610.

[3] Li, J., Socher, R., & Hoi, S. C. (2020). Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394.
clarity,_quality,_novelty_and_reproducibility: $\textbf{Clarity}$: This manuscript is well-written and clearly structured, and the ideas are easy to follow and well-presented.

$\textbf{Quality}$: This manuscript is of good quality.

$\textbf{Novelty}$: While the core label-noise learning method is not novel [1], the authors still manage to make other non-trivial contributions, such as estimating open-set noisy label distributions under federated learning settings, and a clear formulation of the problem, pointing out the possible weakness of existing methods. I would say the novelty of this manuscript is enough for an ICLR paper.

$\textbf{Reproducibility}$: Authors released their implementation code with instructions for reproduction, I believe the results are reproducible.

[1] Liu, Y., & Guo, H. (2020, November). Peer loss functions: Learning from noisy labels without knowing noise rates. In International conference on machine learning (pp. 6226-6236). PMLR.
summary_of_the_review: Overall, this manuscript studies an important and well-defined research problem and proposed a theoretically sound solution. However, the experiments should be more comprehensive to make their contribution more convincing, and whether their DP labels are privacy-preserving given DP transition matrix is debatable. I tend to reject this manuscript under its current version, but I am also willing to adjust my score upon a satisfactory and convincing rebuttal regarding my questions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a federated learning method under openset noisy labels. The idea of this paper is from the study of peer loss in centralized training with noisy labels. With global DP matrix, peer loss function can be computed and learn ML models with noisy models. To employ the peer loss in federated learning, the proposed method generates and broadcasts $T_{DP}$ to each clients to compute the estimated global peer loss. From the experiments and ablation studies, the proposed method performs better on datasets with synthetic label noises as well as real-world label noises.
strength_and_weaknesses: Strength:
+ The idea in this work sounds interesting, which shares a global transmitted matrix in each clients to obtain the posterior label distribution for peer loss computation.

+ From the experiments, the proposed method indeed achieves some good results under label noises, compared with many other pipelines, such as FedAvg, FedProx etc.

Weakness:
- The initialization of transmitted matrix is important. For a real world federated learning system, how to initialize the transmitted matrix, since the data distributions for individual clients are unknown?

- In experiments setup, it is mentioned “The total communication round with the server R is 300 and differential privacy ε are 3.98, 5.98 and 3.95 for CIFAR-10, CIFAR-100 and Clothing-1M, respectively.”. How do those ε be selected?

- From Table 3, the performance is increased with increasing epsilon values. However, the best performance is obtained when 3.98 is set. 3.98 is very close to 4, which provides much worse performance. The results look unreasonable. Do you have any intuition or explanation for the results? Does the proposed method is not stable enough. If so, how do you select an epsilon value?

- In line 14 of Alg2, which loss do you use to measure the difference between predicted label and GT label? Cross entropy loss, or other loss? Does this loss function affect the results ?

- In Alg2, line 2,4,5 use y, while line 12-14 use Y. The same notation should be used.

- In the definition of T_{DP}, what is the meaning of K ? Parameter K is not explained.
clarity,_quality,_novelty_and_reproducibility: The idea of applying peer loss in federated learning is interesting, however, the quality of this paper needs to improve. My main concerns are about the transmitted matrix initialization for real-world applications. The method is easy to reproduce.
summary_of_the_review: This paper presents a new method for federated learning under noisy labels. The proposed method sounds interesting, however, there are several potential issues as discussed in the weakness section. As a result, I vote for "marginally below the acceptance threshold".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper defines a openset noise problem in federated learning, and proposes a corresponding solution FedPeer. FedPeer has two critical stages: 1) The author adopts label differential privacy to share the label space of the client, thereby realizing privacy-preserving global label communication. 2) The authors sample the contrastive labels from the global label distribution obtained in the previous stage, and use the output of the local model to compute the peer loss, enabling robust local gradient updates. The proposed method is validated through quantitative comparison with baselines, on multiple datasets (CIFAR-10, CIFAR-100, Clothing-1M, CIFAR-N).
strength_and_weaknesses: Strength:
1. Paper is straightforward and easy to follow, problem is compelling and ideas communicated clearly.

2. Experimental analysis and comparison presented show that the method is useful.

Weaknesses:

1. The openset noise problem defined in this paper is that the observed label space at these clients may be differ as well, even though their underlying clean labels are drawn from the same label space. How is this problem different from the label noise problem in non-iid federated learning scenarios with skewed label distributions?

2. The previous description of the problem is relatively clear, and the example in section 3.2 can be placed in the appendix.

3. Whether the server can use the collected DP label distribution to estimate the original label distribution through the noise transition matrix, which brings the risk of privacy leakage to some extent.

4. When using the symmetric strategy to synthesize openset label noise, the labels are evenly labeled as the wrong class. Does this weaken the difference in label spaces between clients, and thus deviate from the problem scenario addressed by this paper.

5. From the experimental results, the effectiveness of the method is significant when the noise rate is small, but the method seems to fail when the noise rate is high. How to explain this phenomenon?

clarity,_quality,_novelty_and_reproducibility: 1.	Clear writing and presentation, except for a few spelling errors.
2.	Good quality, since it defines a openset noise problem in federated learning.
3.	Limited technical novelty in method design.
4.	The method is easy to reproduce.

summary_of_the_review: Although I think the openset noise problem in federated learning as defined by the authors is interesting and worth discussing in the research community. However, in the method design of FedPeer, the authors designed two stages “Privacy-preserving global label communication” and “Peer gradient updates”, in which both stages directly adopted existing methods and lacked novelty. Therefore, at this stage, I will vote for marginally below the acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was generally appreciated for practical applicability of the proposed approach (openset noisy labeled data and federated learning), but serious remarks have been also raised.
The main issues, among several requests of clarifications for some stages of the proposed method, regard the novelty aspects/incremental flavour, the need of improving experimental analysis - also in comparative terms - and to justify better the results, and, especially, the risk of information/privacy leakage with respect to the Federated Learning assumptions. 
Authors rebuttal and discussion were raised over general and specific issues, but in the end all ratings remained below threshold, making this paper not acceptable for publication to ICLR 23.
Decision: Reject