Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies robust learning on instance-dependent label noise. It proposes to discover the noise transition matrix and identify clean labels at the same time during training. It also proposes a metric to measure the probability of a label being correct, which is based on historical predictions of an instance during training. Such a metric can be used to better identify both correct labels and noisy labels, which facilitates the learning of noise transition matrix and clean labels.
strength_and_weaknesses: Strength:
* The paper is overall organized well
* Introduction to the related work is thorough and informative
* Learning the transition matrix and clean classifier simultaneously seems to be an interesting idea

Weaknesses:


**Clarity:** I believe overall this paper is not well-clarified, which greatly hinders reading and comprehension. I'll list some of the clarity issues below for reference.
* First, the background section is not sufficient in introducing the terms. For example, terms like "pseudo label", "catastrophic forgetting", "corrupted labels" in section 4.1 are never defined. I am having a hard time understanding this section without a clear definition of these terms. Most of the time I have to guess their meanings based on my own background knowledge.
* What does Equation (2) mean? I guess the point here is to prove the change of loss is small for high-TCP examples, but why this is important? I think the proposed metric is not based on losses. 
* Quote, "instances with clean labels are mutually consistent with each other in producing gradient updates". This sentence is frequently mentioned in the paper, e.g. Section 3, 4.1, and thus I believe it should be important. But what does this mean? I am expecting some formula to clarify this but there is no. I might miss something here.
* Figure 2 shows the proposed metric is good at identifying correct labels when there is label noise. But how does this label noise generated? The authors only mentioned "manually add IDN at 0.4 noise rate (IDN-0.4) onto a benchmark dataset CIFAR10", but I didn't find any details of this, even in the appendix. I think such detail is critical here, as we would expect the proposed metric not to specialize on some particular type of label noise. Experiments on most label noise settings here can also be helpful but I didn't find any figures similar to Figure 2, but experimenting on different label noise settings.

**Quality**: There are quite a few unjustified premises in this paper, which cause difficulty for one to judge the contribution. For example, 
* In section 3, quote, "Apparently, at the instance level, the one-hot prediction of an instance is a more robust metric than the loss because the former has a tolerance to the change of predicted class posterior while the latter has not". But this doesn't appear apparent to me. I can also justify an opposite claim in a similar way, namely the loss considering the entire probabilistic prediction thus is more robust.
* In Section 4.2, quote, "high noisy-TCP instances naturally indicate the instance is learned better and faster for predicting y, leading to stable and fast learning.". But I cannot see the reason here. Why is this premise natural?
* I also have some questions about the experiment results. In Table 1, why baselines such as Dividemix are missing? Is it because Dividemix is not applicable here? In Table 2, why a new method is introduced (TCP-D)? What about the performance of the original method? And there are very few descriptions of this new method.

**Novelty**:
* I believe the proposed metric itself is not novel. Using prediction stability to identify correct or noisy labels is seen in previous works, e.g. [1].
* The idea that using a designated model to learn the transition matrix is relatively novel to me. But apart from one sentence in the abstract, I didn't find any further explanation or justification for this idea.

[1] An Empirical Study Of Example Forgetting During Deep Neural Network Learning. Toneva et al.



clarity,_quality,_novelty_and_reproducibility: As mentioned in detail above, the clarity and quality of this paper may be questionable. The novelty may be limited.
summary_of_the_review: I believe there are some interesting ideas in this paper. But right now, the clarity issue makes it almost impossible to judge the novelty and contribution. I think significant revision is required to improve the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel metric, Time Consistency Prediction (TCP), as criteria to guide curriculum learning for jointly training the label transition matrix and clean classifier.
strength_and_weaknesses: Strength:

1. The proposed metric TCP for curriculum learning is theoretically motivated. In particular, empirical results show that it is a very interesting alternative that outperforms several existing loss based selection criteria.
2. Integrating the TCP, Algorithm 1 is a nice general framework to combine both curriculum learning and label noise transition learning. Empirical experiments are quite strong.

Weaknesses / Major Concerns:

1. Regarding the motivation of TCP, in particular TCP vs. time consistency of loss (TCL), I find it a bit surprising that TCL works so much worse than TCP (Fig 2). The authors explained that 1-hot prediction is more robust than loss at instance level (second paragraph of Sec 3), but the authors later also said clean-TCP instances' loss change can be bounded with a very small value (left to Fig 3). May the authors give more justifications?
2. When motivating the curriculum learning in Sec 4, Fig 3 sees a drastically decreasing clean ratio of original noisy labels, while Fig 2 sees TCP clean ratio still stay high towards the end of training. Why is this? Is this due to the linearly increasing curriculum size in Fig 3? There is some prior work that shows fixing the number of instances being selected given small losses is probably not a good idea (e.g. https://arxiv.org/pdf/2104.02570.pdf), should consider setting a (dynamic) loss threshold instead.
3. Continuing on the previous question, how is the number of selected instances decided at each epoch in the experiments (e.g. linear/exponential growth as done in Fig 3/4)? Is there a way to dynamically control the number of selected instances with TCP?
4. Despite that the synthetic data experiments look very promising, the authors decide to use a variant of TCP-D in the real world data experiments (Tab 2). TCP-D extends DivideMix by selecting high noisy-TCP data to learn the transition matrix and using it to fine-tune the whole data. Given my understanding of the contribution of the paper, would it make more sense to actually replace the GMM-based sampling selection part of DivideMix by TCP guided curriculum? Why the particular choice of TCP-D? Did the authors try other extensions of DivideMix inspired by TCP?

Minor comments:

1. Should the authors include EMA over loss (Zhou et al. 2020) as it has been shown to be more robust than instantaneous loss, since TCP also considers the entire training history?
2. I think TCP w/o D_t in Tab 3 performs well enough to be standalone methods given that they are much simpler alternatives to the full model without the whole label transition or extra fine tuning on entire data (pending more results?)
clarity,_quality,_novelty_and_reproducibility: The paper discusses a novel and empirically strong metric for curriculum learning that challenges the widely used small-loss assumption. Many readers in the field should find the work interesting. The paper is well written.
summary_of_the_review: I believe the novelty and contribution of the paper outweigh the weaknesses of current theories, results and writing. I'm happy to increase my score if the authors can address some of my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for training a classifier on data with Instance-Dependent Noisy Labels (IDN Labels, i.e. where label noise is a function of the instances themselves).  Their technique is within a class of techniques that monitor a heuristic during training that measures some notion of noise in labels, and then using instances selected as "clean" and "noisy" then learn a transformation matrix that describes the relationship between true and noisy labels in an instance-dependent manner.  The main novel observation is the the heuristics in prior work are sensitive enough that monitoring them from one training epoch to another doesn't provide clear signal  as to which instances and "clean".  They instead use a TCP metric that is a running average of whether the predicted label has changed between gradient updates.  They show that if you assume that their metric is relatively stable, then their curriculum learning approach that introduces new instances during training does not result in catastrophic forgetting.  Empirically, the proposed method outperforms a number of baselines, both with real and synthetic noise.
strength_and_weaknesses: Strengths
1. Clearly techniques that learn from instance-dependent noise have considerable practical applications.
2. Their approach is relatively simple and seems like it could be implemented easily with common machine learning packages.
3. In terms of classification accuracy, the proposed approach achieves impressive performance relative to baselines.

Weaknesses
1. The paper suffers from poor organization which makes its somewhat difficult to follow.  Namely, there needs to be a self-contained "preliminaries" before describing the techniques that are introduced.  As it is written right now, lots of notation and leveraged prior work are introduced throughout the paper.  As a result, it is both difficult to follow the notation and difficult to identify what specifically was novel about the proposed techniques.  I recommend taking a step back and thinking about how to reorganize the paper entirely, and also consider discussing the most related IDN label learning algorithms in a more formal way to establish them as a point of reference to the proposed new contributions made by this work.
2. The claim of the proposed technique being immune to catastrophic forgetting is discussed as a formal contribution, but it neither formally defined nor strongly argued.  There needs to be a formal statement of what is being proved.  Also, the proof hinges on the statement: "Since x′ is selected with high clean-TCP... has been verified in Figure 1".  This is not a formally proved statement, which makes the "proof" more of an empirical observation.
3. The curriculum learning algorithm requires training two models to completion before training begins.  It is not clear how computationally expensive this is relative to the baselines.  To provide a more fair understanding of the performance of the proposed technique, I think wall run times should be provided for the training of the models in the empirical results section.

clarity,_quality,_novelty_and_reproducibility: Clarity - Overall the paper could use major revisions to organization of the paper as well as a few passes to correct grammatical errors.  Because of the shortcomings in clarity, the paper is not easy to read despite the techniques being relatively simple.

Quality - The authors make a strong attempt at providing intuition and empirical justification for their proposed method.  They also review a large amount of prior work. Presentation, organization, and clarity issues aside, I believe the work is of moderate to high quality.

Novelty - The main novel contribution is the TCP heuristic used to select clean and noisy instances.  The heuristic is empirically argued for, but no strong theoretical analysis is provided.  As such, the paper, while empirically successful, is not very novel.

Reproducibility - Mainly because a number of baselines are not fully discussed nor is code provided, I think it would be rather difficult to reproduce the results of this work.
summary_of_the_review: While the clarity of the paper is a barrier for understanding and the proposed technique is not of significant novelty, the authors provide strong intuition for their approach and follow that with strong empirical performance.  Because this technique seems to be the empirical leader for the IDN Labels learning problem, I am leaning towards acceptance.  I do think for a camera ready copy, the authors should take a fine-grained critical look at the writing and organization of the paper to ensure that the main contributions are more clearly articulated.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work targets to the instance-dependent noisy-label learning problem and proposes a new curriculum learning-based algorithm. In particular, the authors calculate the 'time-consistency of prediction' (TCP) scores that indicate how consistent are the predictions of samples over the course of training. Then, the proposed framework selects anchor instances by the TCP scores and estimates the transition matrices. Experiments show the proposed method achieves better performance than the baselines. 
strength_and_weaknesses: Pros:

1. This work provides a different perspective on selecting clean examples for noisy-label learning by the consistency of predictions, which brings some insights to the community. 

2. The authors provided thorough analytical results of the TCP scores.

Cons:

1. My first concern is that the proposed TCP score, while seemingly new, may be analogous to the pseudo-labeling procedure of semi-supervised learning (SSL) and inherently not a new method. In SSL, methods like FixMatch clip the model prediction on unlabeled data as their pseudo-labels for training; Pi-Model/Mean-Teacher adopt a time-consistent prediction as pseudo-labels. When the model is confident to one instance (for example, max prediction > 0.95), its prediction would also be stable as well and has TCP scores. Such a pseudo-labeling method had also adopted in the noisy-label learning regime [1]. In fact, I doubt TCP is inherently a semi-supervised learning algorithm and may not be better than such a pseudo-labeling strategy. 

2. The relationship between the TCP score and the instance-dependent noise is unclear. Why is TCP particularly important to the ID setup? In effect, as the proposed method also integrates the SSL techniques, I believe simple pseudo-labeling can also accomplish the anchor sample selection procedure for transition matrix estimation. 

3. The empirical study is weak for me. 

    3.1 First, the real-world noisy label CIFAR dataset has a version of CIFAR-100-N, but the results are reported in neither Table 2 nor Appendix. Does the proposed method underperform on this dataset? 

    3.2 Some important baselines are missing. For example, the UNICON, and SOP methods have shown promising results. The seminal DivideMix method is not compared in Table 1. While it's not mainly designed for ID NLL, I believe it can also handle this task empirically. Notably, TCP might be a semi-supervised NLL method and simply comparing it with naive NLL baselines is unfair.

    3.3 Second, the ablation study is really weak. The authors should thoroughly analyze the effectiveness of different components. Does the estimated transition matrix really work? What if we directly run a dividemix model with TCP-selected samples? Can we replace the TCP-based selection with a pseudo-label-based selection for the transition matrix? 

    3.4 The ablation is mainly conducted on CIFAR-10, which is not convincing. 

4. I didn't find the definition of \hat{y}, which made me hard to understand Eq. (1). 

[1] Li J, Xiong C, Hoi S. MoPro: Webly Supervised Learning with Momentum Prototypes[C]//International Conference on Learning Representations. 2020.

[2] Karim N, Rizve M N, Rahnavard N, et al. UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 9676-9686.

[3] Liu S, Zhu Z, Qu Q, et al. Robust Training under Label Noise by Over-parameterization[J]. ICML, 2022.
clarity,_quality,_novelty_and_reproducibility: This work is overall clear and easy to follow, but some definitions are missing. The proposed method is inherently similar to thresholding-based pseudo-labeling/correction methods in noisy label learning [1] and the originality is doubtful. 
summary_of_the_review: The originality is doubtful and the empirical study is weak. So I vote for rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper discusses the issue of noisy labels in the train data, in particular when this noise can be dependent on the input. The paper motivates its approach based on observations about the difficulty of estimating a noise transition matrix, and creates a technique to monitor consistency of predictions during gradient updates. The experimental results are quite strong, though there might have I think been more work done on toy problems to demonstrate the approach better. I think the reviewers had some issues with the clarity of the paper, which I agree with and post rebuttal the review scores generally remained below threshold. There are also some statements that I find difficult to accept -- for example, in contrast to what the authors state, learning the noise transition doesn't require explicit knowledge of the clean label. There are works that treat the clean label as an unknown latent variable and, based on an assumption that noise will generally have a higher chance (for binary labels) of staying in the correct class, this allows the noise process to be identified. I think some of the basic explanations and assumptions in this work are not fully clear or justified. In general, it is also quite strange (to me at least) why a temporal criterion is used to fix an issue (label noise) that has itself nothing to do with learning dynamics. 
justification_for_why_not_higher_score: There is no champion for this paper and I personally don't think this is good enough work. It's confusing and unclear in places.
Decision: Reject