Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies how to leverage self-supervised learning features to improve the estimation of the transition matrix. The paper developed a framework for learning the instance-dependent transition matrix. The framework is composed of confident examples selecting stage using contrastive learning and constraint transition matrix revision stage. The contrastive co-selecting stage leverages different degrees of data augmentations to acquire representations with semantic information correlating with clean labels. By jointly optimizing noisy training sample loss and clean confident sample loss, Constraint transition matrix revision improve the estimation of the transition matrix and classification accuracy.
strength_and_weaknesses: Strength: 

1. This paper provides a novel understanding of how self-supervised learning representation can help to construct a model with noisy label data. It is an interesting discovery that SSL representation can help learning with noisy labels. The idea that contrastive learning representations contain some information about clean labels is intuitive and novel.

2. This paper also provides a novel understanding of how to jointly optimize loss function on both clean labels and noisy labels to helps estimate transition. 

Weakness: 

1. The co-selecting stage is similar to a combination of the ensemble learning method and pseudo-label generation. It is unclear why training two classifier heads with different data augmentations instead of different contrastive models, which may help decrease model-related errors?

2. The authors claim that the method can achieve state-of-the-art performance on different datasets, but more datasets should be introduced, such as CIFAR-100N and WebVision to show if the methods still work when the number of classes and the transition matrix become larger.

3. The experiment results of baselines are significantly different from those reported in other papers (see below). It would be better to provide the codes and models used in the paper or point out how the experiment setting can be different from previous works.

Xia X, Liu T, Han B, et al. Part-dependent label noise: Towards instance-dependent label noise[J]. Advances in Neural Information Processing Systems, 2020, 33: 7597-7610.

Yang S, Yang E, Han B, et al. Estimating instance-dependent bayes-label transition matrix using a deep neural network[C]//International Conference on Machine Learning. PMLR, 2022: 25302-25312.

clarity,_quality,_novelty_and_reproducibility: This work provides a novel understanding of self-supervised learning and optimization while the novelty in the proposed approaches could be better clarified. 
summary_of_the_review: This paper developed a novel framework for learning the instance-dependent transition matrix by utilizing a self-supervised learning. The idea is novel and interesting, while more clarifications in the novelty in the approaches, the comparison experiments and experiments setup. The accuracy of the experimental results should be more convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a method for estimating instance-dependent transition matrix using contrastive pretraining. The method starts from the pretrained network using contrastive loss, and train a set of classifiers to divide data into confident set. They are further used to refine classifier, transition matrix, as well as the backbone. In experiments, the proposed method is tested on standard noisy-label learning benchmarks, demonstrating improved performance over existing works.
strength_and_weaknesses: * Strength
  - The paper tackles important problem of transition matrix estimation using self-supervised pretraining.

* Weakness
  - The paper is not well written. 
  - Overall, the `contrastive` is used only to provide pretrained backbone, so it is unclear why the method is to be called a contrastive noisy-label learning. In theory, the pretrained backbone can be replaced with any networks.
  - How much performance improvement is attributed to the use of pretrained model and the use of fine-tuning? While the use of contrastive pretraining is useful in practice, it does not add much scientific value to the community. For example, a fair comparison to previous works would be to employ the same pretrained backbone and evaluate their methods, as some previous works are agnostic to the network initialization (e.g., BLTM, NPC).
  - How does method compared against more recent works, such as [DivideAndMix](https://arxiv.org/pdf/2002.07394.pdf)?
  - In Equation (2), are there any constraints that makes T's a transition matrix? It is unclear what it means by the transition matrix (e.g., what's the difference between transition matrix and a matrix?) and how it is regularized to learn a `proper` (which is not defined) transition matrix.
  - In section 3.2., the use of validation accuracy in selecting the best transition matrix is problematic. How large is the validation set? Authors also need to provide ablation w.r.t. the size of validation set and its impact on the final performance.
  - In experiment, it is unclear how `CONL-NR` model is trained. Does it mean the model is trained with the loss in Equation (7) but without updating the parameters w.r.t. transition matrix?

* Misc
  - The paper requires significant revision as there are numerous grammatical errors, inconsistent notations, etc. To list a few:
    - There are many broken sentences, e.g., pg 3, `motivated by the success of ... classification task.`; pg 5, `Once we have the confident sample, ..., clean labels.`; 
    - Equation (6), $\tilde{y}$ is not used.

clarity,_quality,_novelty_and_reproducibility: * The paper is not very well written. Besides the grammatical errors, the paper does not provide much insight on the proposed method beyond the use of pretraining and empirical results. 

* Novelty is limited. Specifically, the use of pretrained network is a usual practice. The use of confident / unconfident samples to learn from noisy labels is not new, e.g., [DivideAndMix](https://arxiv.org/pdf/2002.07394.pdf), which is not compared against.

* Overall method seems straightforward, though there are many missing implementation details (e.g., hyperparameter settings).
summary_of_the_review: The use of pretrained network for image classification is well established, so it is not surprising if it improves when applied to label noise transition matrix estimation problem. Unfortunately, the study is not well conducted and it is hard to understand which part of the proposed framework contributed the most to the empirical success. Additional study to highlight the effectiveness of transition matrix learning should be provided.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A novel self-supervised learning framework is proposed for noisy labels and can effectively improve the accuracy of instance-dependent transition matrices. Specifically, this method mainly contributes to two parts. First, a contrastive co-selection method is proposed to select samples with high confidence. This method obtains the basic feature extraction backbone network learned by MOCO, and then design criterions to select confident samples. Secondly, a constraint T-revision method is proposed, which uses confidence samples to fine-tune T(x) to improve the classification accuracy.
strength_and_weaknesses: In this paper, self-supervision is used to improve the classification performance of noisy-label samples. And mining the information of the confident samples to improve the performance of the model. The method is simple and effective.


1. Use self-supervision to improve the estimation of the transition matrix as described in the title of this paper. But the core idea of this paper is not the design of self-supervised algorithms. The self-supervised MOCO algorithm is only a pre-training method as the first step of the method, and other pre-training methods can also be used in this part. In addition, although this algorithm trains two classifiers using the data with strong and weak augmentation transformation. However, these two branches use noisy label training, and the two-branch training process does not have the idea of self-supervision. Therefore, I do not think that the algorithm is improved by self-supervision as described in the title.
2. The innovation of this paper is weak. The idea of training two classifiers to be complementary has been extensively studied in papers from earlier years. The designed contrastive co-selecting method utilizes two criterions for sample selection, which is not very novel. At the same time, the improvement of constraint T (x) revision only uses the confident samples based on the T-revision algorithm. The algorithm has less theoretical support.
3. The abstract part does not clearly show the specific method. It is necessary to outline the innovative idea of the algorithm in the abstract, so that readers can quickly understand the core idea of the paper. The summary part has the same problem and needs to explain the advantages of the method, the direction of future improvement, etc.
4. Figure 1 of this paper describes the workflow of the method. However, the figure is relatively rough, which does not allow readers to quickly understand the modules of the algorithm and the specific execution processes.

clarity,_quality,_novelty_and_reproducibility: Clarity: The logic of the algorithm description in this paper is relatively clear, and the text part and formula can clearly introduce the details of the algorithm. However, the abstract part needs to be further refined so that readers can quickly understand the content of the paper. The picture of the algorithm work flow still needs to be adjusted for readability.
Quality: The quality of this article is good. The description of the method is clear, the experiments demonstrate the greatly improved on multiple data sets, and can still be extended to justify the performance of the algorithm
Novelty: This algorithm has certain innovation. The algorithm jointly selects the confident samples in the two branches of the design as the overall framework, which can effectively improve the accuracy of noise label classification.
Reproducibility: The algorithm can be reproduced according to the description in this paper. But the threshold of the two criteria is not provided. The description of some hyperparameters needs to be described more clearly.

summary_of_the_review: This paper achieves a large performance improvement through the proposed contrastive co-selecting and constraint T (x) revision methods. We think this paper has certain innovation. However, the core innovation points of the article and the description of the title abstract are not appropriate. In the experimental part, the method will be more convincing if it can be verified with other methods (DivideMix, etc.) on large-scale real-world datasets (Clothing1M and WebVision, etc.).
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposed a novel method for estimating the instance-dependent transition matrix. The self-supervised learning are employed to mine the clean samples. And then refines the transition matrix estimated by T-Revision via employing the selected confident examples. The empirical results varifies the effectiveness of proposed method.
strength_and_weaknesses: This paper refines the learned transition matrix by employing the selected confident examples, which is interesting. However, there exist some concerns as follows.
- The idea is novel, while the formulation is not rigorous. Specifically, why use self-supervised learning to select clean samples? What is the difference with other sample selection methods. 
- The self-supervised learning has help improve the label noise learning, e.g., C2D [1], Sel-CL [2]. It is fair to epuip compared methods with contrastive learning. It can be seen that CoNL (w/o MoCo) in table 9 & 10 evidently underforms compared methods. From this viewpoint, I doubt the effectiveness of proposed method, since the performance increase completely stem from self-supervised learning.

[1] Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, AlexMBronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1657–1667, 2022.

[2] Li S, Xia X, Ge S, et al. Selective-supervised contrastive learning with noisy labels[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 316-325.
- Also, the method uses different data augmentation startegies. It has been found that using two separate pools of augmentation operations for two separate tasks is beneficial for noisy label learning in [3]. I doubt that the performance increase mainly dues to such learning manner, rather than the proposed method. 

[3] Nishi K, Ding Y, Rich A, et al. Augmentation strategies for learning with noisy labels[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 8022-8031.

- Eq.(7) is not proper for transition matrix. The transition matrix is used for the whole training dataset, rather than just only noisy datasets. This deviates the classifier consistent theory of  transition matrix.

- The method only ran experiments on several simple datasets. The datasets with more classes, more complex noise structures, and real-world datasets, e.g., WebVision, are necessary to demostrate the effectiveness of proposed method.

- How to guarteen that $T_{\zeta}$ does not overfit the noisy labels? And the network structure of transition matrix is not clear. How to guarteen that network can estimate more classes?

clarity,_quality,_novelty_and_reproducibility: It’s interesting to combine contrastive learning and transition matrix estimation. But the mechanism of learning transition matrix by noisy labels is not clear. And the effectiveness of proposed method is not properly justified. The experiment is divided into several stages and contains many hyperparameters, so it is not easy to reproduce.
summary_of_the_review: This work proposes an interesting perspective to combine self-supervised learning and instance-dependent transition matrix.  While the mechanism of learning transition matrix by noisy labels is not clear. The empirical results can not support the  effectiveness of proposed method, especically some additional techniques, which potentially bring much performance increase. Overall, the writting is poor, and the noverty and effectiveness of proposed method need to further clarify.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new self-supervised learning method for estimating the instance-dependent transition matrix. While this problem is interesting, the authors raised a series of concerns, including the formulation is not rigorous and the contributions are not sufficiently novel. There is still insufficient interest in accepting this paper after rebuttals.
justification_for_why_not_higher_score: Some flaws in this paper that were spotted by the referees should be addressed before acceptance.
Decision: Reject