A Time-Consistency Curriculum for Learning from Instance-Dependent Noisy Labels
Authors: Songhua Wu
Tianyi Zhou
Yuxuan Du
Jun Yu
Bo Han
Tongliang Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Many machine learning algorithms are known to be fragile on simple instance-independent noisy labels. However, noisy labels in real-world data are more devastating since they are produced by more complicated mechanisms in an instance-dependent manner. In this paper, we target this practical challenge of  \textit{Instance-Dependent Noisy Labels} by jointly training 
(1) a model reversely engineering the noise generating mechanism, which produces an \textit{instance-dependent mapping} between the clean label posterior and the observed noisy label; and (2) a robust classifier that produces clean label posteriors. Compared to previous methods, the former model is novel and enables end-to-end learning of the latter directly from noisy labels. An extensive empirical study indicates that the time-consistency of data is critical to the success of training both models and motivates us to develop a curriculum selecting training data based on their dynamics on the two models' outputs over the course of training. We show that the curriculum-selected data provide both clean labels and high-quality input-output pairs for training the two models. Therefore, it leads to promising and robust classification performance even in notably challenging settings of instance-dependent noisy labels where many SoTA methods could easily fail. Extensive experimental comparisons and ablation studies further demonstrate the advantages and significance of the time-consistency curriculum in learning from instance-dependent noisy labels on multiple benchmark datasets.

Under review as a conference paper at ICLR 2023
A TIME-CONSISTENCY CURRICULUM FOR LEARNING FROM
NOISY LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
Many machine learning algorithms are known to be fragile on simple instance- independent noisy labels. However, noisy labels in real-world data are more dev- astating since they are produced by more complicated mechanisms in an instance- dependent manner. In this paper, we target this practical challenge of Instance- Dependent Noisy Labels by jointly training (1) a model reversely engineering the noise generating mechanism, which produces an instance-dependent mapping be- tween the clean label posterior and the observed noisy label; and (2) robust classi- fier that produces clean label posteriors. Compared to previous methods, the former model is novel and enables end-to-end learning of the latter directly from noisy labels. An extensive empirical study indicates that the time-consistency of data is critical to the success of training both models and motivates us develop cur- riculum selecting training data based on their dynamics on the two models' outputs over the course of training. We show that the curriculum-selected data provide both clean labels and high-quality input-output pairs for training the two models. There- fore, it leads promising and robust classification performance even in notably challenging settings of instance-des deetanceddeeedeent noisy labels where many SoTA methods could easily fail. Extensive experimental comparisons and ablation studies further demonstrate the advantages and significance of the time-consistency curriculum in learning from instance-dependent noisy labels on multiple benchmark datasets.
INTRODUCTION
The training of neural networks can easily fail in the presence even the simple instance-indepeedent noisy labels since they quickly lead to model overfitting of the noises (Zhang et al., 2017). In practice, however, it is usually challenging to control the labeling quality of large-scale datasets because the labels were generated by complicated mechanisms such non-expert workers (Han al., 2020b). An average of 3.3% noisy labels is identified in the test/validation sets of 10 of the most commonly-use datasets in computer vision, natural language, and audio analysis (Northcutt al., 2021). Moreover, real-world noisy labels are generated in an instance- -ependen manner, which is significantly more challenging address than the most widely studied but oversimplified instance-independent noises, which assume that the noise only depends on the class (Wei et al., 2022).
Two principal methodologies have been developed to address the label noises: (1) detecting samples (X, Y) with correct labels Y Y (empirically, they are the ones with the smallest loss values) and using them to train a clean classifier (Han et al., 2018b; Yu et al., 2019); (2) learning the noise generating mechanism, i.c., a transition matrix T defining the mapping between clean label and noisy label Y such that P(Y X) = PT X), where P (- X) denotes the posterior vector, and then using it to build statistically consistent classifiers (Liu & Tao, 2016; Patrini et al., 2017; Yang et al., 2021). Although both methodologies have achieved promising results the simplified instance- independent (class-dependent) setting, they have non-trivial drawbacks when applied to the more practical but complicated instance-dependent noises: (1) the "small loss" trick no longer effective in detecting correct labels (Cheng et al., 2021) because the loss threshold drastically varies across instances and is determined by each transition matrix T(X): (2) the instance-dependent transition matrix T T(X) is not identifiable given only the noisy sample and it heavily relies the estimation of clean label Y in the triple (X, Y, Y, (Yang et al., 2021), which is an unsolved challenge in (1).

---

Under review as a conference paper at ICLR 2023
Therefore, the two learning problems are entangled, ie.,, the training of a clean label predictor and the transition matrix estimator depends on each other's 's accuracy, which substantially relies on the quality of training data (X, Y, Y) Specifically, the "small loss" trick cannot provide a high-quality estimation of Y due to the instance-specifie threshold of loss. Moreover, the estimation of Y can change rapidly due to the non-stationary loss, which can fluctuate during training and provide inconsistent training signals over time for both models if selected for training. Furthermore, the data subset selection inevitably introduces biases toward easy-to-fit samples and degrades the data diversity (Yang et al., 2021: Cheng et al., 2021; Berthon et al., 2021; Cheng et al., 2020), which in fact is critical to the training and the accuracy of both models, especially the transition matrix estimator, because easy-to-fit samples usually have extremely sparse transition matrices.
To tackle the above issues, we propose novel metric Hime--Cosisstncy of Prediction (TCP)" to select high-quality data train both models. TCP measures the consistency of model prediction for an instance over the course of training. which reflects whether its given label results in gradients consistent with the majority of other instances, and this criterion turns out to be a more reliable identifier of clean labels. When applied to the training of clean label predictor, TCP is more accurate in clean label detection than "small loss" " (or high confidence) criterion, because it avoids the comparison of confidence for samples with instance- neseanceddeenneet loss/confidence thresholds. Moreover, when applied to the training of the transition matrix estimator, TCP measures the cime-consistency of predicted noisy labels Y. Surprisingly. it also faithfully reflects the correctness of the predicted clean label Y. Since the objective to estimate the transition matrix is defined by both Y and Y, selecting samples with high TCP considerably improves the training of the transition matrix estimator. In addition, to exploit the data diversity in training the two models, we apply a curriculum that starts from selecting only few high TCP data for early-stage training but progressively includes more training data once the two models become maturer and more consistent.
In this paper, we develop a three-stage training strategy with the TCP curriculum embedded. In every training step, we first update the clean label predictor using selected data with high TCP on this model, followed by training the transition matrix estimator given the predicted clean label posterior and the noisy labels on selected data with high TCP on the estimator, and end with fine-tuning the clean label predictor directly using the noisy label and the estimated transition matrix. It is worth noting that the TCP metrics for the two models are updated using the model outputs collected from this dynamic training process without causing additional cost. As demonstrated by extensive empirical studies and experimental comparisons, our method leads to efficient joint training of the two models that mutually benefits from each other and produces an accurate estimation of both the clean label and instance-dest dependent transition matrix. On multiple benchmark datasets with either synthetic or real-world noises, our method achieves state-of-the performance with significant improvements.
2 BACKGROUNDS AND RELATED WORKS
Let (X E X cl. c} be the random variables for instances and clean labels, where X represents the instance space and c is the number of classes. In many real-world applications, the observed labels are not always correct but contain some noise. Let Y be the random variable for the noisy label. What we have is a sample { ......, ... (z,, }n.} drawn from the noisy distribution D, of the random variables (X, Y). We aim to learn a robust classifier that could assign clean labels to test data by exploiting the sample with noisy labels.
Label noise models. Currently, there are three typical label noise models, which are the random classification noise (RCN) model (Biggio et al., 2011; Natarajan et al., 2013; Manwani & Sastry, 2013). the class-dependent label noise (CDN) model (Patrini et al., 2017: Xia et al., 2019: Zhang Sabuncu, 2018), and the instance-dependent label noise (IDN) model (Berthon et al., 2021; Cheng et al., 2021). Specifically, RCN assumes that clean labels flip randomly with a constant rate; CDN assumes that the flip rate only depends on the true class; IDN considers the most general case of label noise, where the flip rate depends on its instance. Since IDN is non-identifiable without any additional assumption, some simplified variants were proposed. (Xia et al., 2020b) proposed the pert-dependent label noise (PDN) model which assumes that the label noise depends on parts of instances. (Cheng et al., 2020; Yang et al., 2021) assume that the flip rates are dependent instances but can be upper bounded by a value smaller than 1. This paper focuses on the original IDN model without introducing any additional assumptions.
2

---

Under review as a conference paper at ICLR 2023
Estimating the transition matrix. The structure of label noise is usually formulated by a transition matrix T, where is the number of classes and its ij-th element (z) P(Y j X = z), which represents the probability that the instance x with the clean label Y = i actually has a noisy label y = j. The transition matrix naturally establishes the connection between noisy posterior and clean posterior, i.e., P( x) = (T)) (Y X), and thus plays an important role in building statistically consistent classifiers in label-noise learning (Liu & 'Tao, 2016; Scott, 2015). To estimate it, a cross-validation method can be applied for the binary classification task (Natarajan et al., 2013). For CDN, the transition matrix could be learned by exploiting anchor points (Patrini et al., 2017; Yu et al., 2018). For IDN, the transition matrix for an instance could be approximated by a combination of the transition matrices for the parts of the instance (Xia et al., 2020b) or a Bayes label transition matrix (Yang et al., 2021). Yao et al. (2021) exploited the causal graph to estimate the transition relations between clean and noisy labels.
Curriculum learning. Curriculum learning was first proposed by Bengio et al. (2009), which describes learning paradigm in which a model is learned by gradually introducing samples of increasing hardness to training. Its effectiveness has been empirically verified in wide range of applications, e.g., computer vision (Chen & Gupta, 2015), natural language processing (Turian et al., 2010), and multitask learning (Graves et al., 2017). Curriculum for label-noise learning has been also investigated. MentorNet (Jiang et al., 2018) pre-trains an extra network producing a data-driven curriculum selecting data instances to guide the training. When the clean validation data is not available, MentorNet has to use a predefined curriculum. RoCL (Zhou et al., 2021) develops a curriculum learning strategy that smoothly transitions between (1) detection and supervised training on clean data; and (2) relabeling and self-supervision on noisy data. Nevertheless, RoCL has no convergence guarantee and needs extra data augmentations to collect spatial-consiseent pseudo labels.
Moreover, existing methods for learning with noisy labels employ heuristics to reduce the side-effect of noisy labels, e.g.. selecting reliable samples (Han et al., 2018b; Yu al., 2019; Wei et al., 2020a; Wu et al., 2020; Xia et al., 2020a), reweighting samples (Ren et al., 2018; Jiang et al., 2018: Ma et al., 2018; Kremer et al., 2018; Reed et al., 2015), correcting labels Tanaka et al. (2018); Zheng et al. (2020), designing robust loss functions Zhang & Sabuncu (2018) 8); Xu et al. (2019); Liu & Guo (2020); Ma et al. (2020), employing side information (Vahdat, 2017; Li et al., 2017), and (implicitly) adding regularization (Li et al., 2021; 2017; Veit et al., 2017: Vahdat, 2017; Han et al., 2018a; Zhang et al., 2018; Guo et al., 2018; Hu et al., 2020; Zhang et al., 2021; Han et al., 2020a).
3 EXAMPLES SELECTION CRITERION:
OF PREDICTION
According to the observation that the loss on instances with clean labels is usually smaller than instances with noisy labels, the loss computed at an instantaneous step has been widely adopted as selection criterion for confident examples (Han et al., 2018b; Yu et al., 2019; Wang et al., 2019). is because instances with clean labels are mutually consistent with each other in producing gradient updates, allowing the model to fit them better and thereby make the loss smaller than instances with noisy labels.
Unfortunately, the instantaneous loss was found only work well on the instance-independde label noise (Cheng et al., 2021). For a deep neural network, because of the non-smooth nature of the loss and the randomness of stochastic gradient descent, the instantaneous loss of each instance can change dramatically between consecutive epochs, leading to huge gap between training sets selected over consecutive epochs. Therefore, it is necessary to take the training history of each instance into consideration. Zhou et al. (2020) proposed robust version of the instantaneous loss as the exponential moving average of it over the course of training. Nevertheless, in the IDN case, each instance with its noisy label is unique pattern, which is more complex and thereby requires a more robust selection criterion. Apparently, at the instance level, the one-hot prediction of instance is more robust metric than the loss because the former has a tolerance to the change of predicted class posterior while the latter has not, i..., the one-hot prediction remains unchanged if the position of the max element in the predicted class posterior vector is maintained but the cross- cross-entropy loss changes once the predicted class posterior changes.
3

---

Under review as a conference paper at ICLR 2023
20
08
fution 12
3X 04
1035
(a) Partitioned at start stage.
(b) Partitioned at early stage.
(c) Partitioned at an end stage.
Figure I: TCP (mean and std.) of three groups (high TCP (10%), middle TCP (80%), and low TCP (10%)) partitioned by the TCP calculated at the start stage (epoch 5), early stage (epoch 30), and end stage (epoch 95) during training ResNet34 on CIFARIO with IDN-0.4 0.4 for 100 epochs.
Inspired by the above insights, we propose time-consistency of prediction (TCP) metric as follows:
1
TC)
t+1 TCP((z)
t+1 InPr+1(2))
(I)
where (n) =1 1 Dt] and Út is the predicted label epoch t. This metric considers
the prediction consistency over the course of training, which can better describe the IDN data and select confident examples than the previous ones. To see this, we first manually add IDN at 0.4 noise rate (see Section. 5) onto a benchmark dataset CIFARIO and train a ResNet34 (He et al., 2016) for 100 epochs with constant learning rate. Since no curriculum strategy is applied here, we select confident examples at every epoch t with fixed number 5,000 according to four types of selection criterion, i.e., instantaneous prediction InP; (x), instantaneous loss f(x), consisens of prediction TCP:(1) and time-consistency of loss (defined in the same way as TCP). Then we count the number of instances with clean labels from the selected confident examples and calculate the clean ratios (ratio of clean instances to selected instances). As shown in Figure 2, can find that the two instantaneous metrics have clean ratios lower than 0.6, which are worse than random selection. As for time-consistenge of loss, the clean ratio is slightly higher than the random selection. Those three metrics are basically not discriminative to the noisy data. By contrast, the proposed TCP metric has distinguishable performance, uplifting more than 20 percent of the clean ratio of the selected confident examples. More empirical study regarding Figures and 2 is provided in Appendix A.
Moreover, we partition the whole data into three groups (high TCP (10%), middle TCP (80%).
and low TCP (10%) )) by the TCP calculated at the start stage (epoch 5), early stage (epoch 30),
0.85
0.80
and end stage (epoch 95). We visualize the mean and variance of the groups through the whole training epochs. shown in Figure 1, the start- stage partition fails at the end stage as three groups are entangled together while the early- stage partition shares the almost same pattern as the end-stage partition. Thus, we can con- clude that the carly-stage TCP is reflective of the property of each instance in the future, which means the time-consistent examples selected in the early stage will not mislead the classifier be- cause their TCP are still high and thus they will still be selected as time-consistent examples in
consistoncy
Time consistency Joss
Iratantan wous Prediction
0.70
Remdem Selection
Instantanecul eous Loss
20
40
60
80
100
Epoch
Figure 2: Clean ratios of the selected top 5000 in- stances ranked by four kinds of instance hardness measures, respectively, during a standard training for 100 epochs. The clean ratio of randomly se- lected instances 0.6 since the noise rate is 0.4. the rest training epochs. Besides, warmup for the TCP proved to be necessary.
TCP GUIDED CURRICULUM LEARNING FOR LABELS
NOISY
Our curriculum contains two sub-curricllum corresponding to two main challenges for solving the IDN problem: (1) detecting examples with clean labels; (2) learning the transition matrix. In this
4

---

Under review as a conference paper at ICLR 2023
section, elaborate how use TCP to design curricula to achieve these goals. Below clean- and noisy-TCP refer time consistency of clean label predictions (arg max P(y x)) and noisy label predictions (arg max P x)) over historical training steps, respectively.
4.1 LEARNING A PRIMARY CLEAN CLASSIFIER WITH HIGH CLEAN-TCP INSTANCES
In Section 3, we demonstrate that clean- TCP can be used to select examples with clean labels because they are mutually consistent with each other producing gradient updates and easy to be learned to have high clean-TCP. However, as shown in Figure 2, not all of the selected high clean-TCP instances have clean labels. The reason why those instances have high clean-TCP but not clean labels is because they are far away from the classification boundary and inherently hard to be misclassified even though their given labels are wrong. Namely, their labels get corrected by the classifier, and high clean-TCP indicates the pscudo labels assigned by the classifier correct.
Therefore, with clean- TCP, we design a curriculum which exploits both examples with clean labels and instances with correct pseudo labels to learn clean classifier. We theoretically prove that introducing high clean-TCP instance with its pseudo label does not cause catastrophic forgetting of the learned confident examples. Consider the situation we have labeled set L (in practice it can be the selected confident examples set) and one unlabeled instance x'. By training on L for one step, have 0c+1= = O: + 1) Ezec Val (I; 0e); and by training on L and x' for one step, have 0.41 02 + 7 (Ergc Val (z; 0z) + Vol (æ'; 0,)), where Og denotes the network parameters at step
I and 1) denotes the learning rate. Then we have
Σ FEC
(z;6(+1) (x: (0x08+1]]
Pitz (x²) Pr (x))
(2)
where pf (a') is the probability of x' belonging to jú at step i, and S is the prediction (pseudo label) of x' at step The detailed derivation is provided in Appendix B. If x' is selected with high cCea p/ (a') very close to Ve
Powurd Label
Selected Number
23000
23000
c.-
Clean
15000
Pri. (a') because it has been verified in Fig- ure I that instances with high clean-TCP in the early stage maintain their high clean-TCP in the future, which means the loss change can be bounded with very small value. Therefore, ex- ploiting high clean-TCC instances with pseudo labels helps to correct corrupted labels and learn clean classifier without causing catastrophic forgetting of the learned examples with correct labels. In Figure 3, we show the clean ratios of the original noisy labels and pseudo labels of instances selected with our curriculum during the whole training process. The clean ratio for pseudo labels maintains amazing high value, much better than the original clean ratio. Therefore, the clean classifier can be learned by minimizing Σ.V L (f vH),, where y* can be the original noisy label or pseudo label in different learning phases. Implementation details can be found in Section 4.3.
c.8
10000
0.7
5000
0.6
00
Figure 3: Clean ratios of selected high clean- TCP examples w.r.t. their original noisy labels and pseudo labels with linear growth of the se- lected number during our curriculum learning on CIFAR10 with 4 for 100 epochs.
4.2 LEARNING A TRANSITION MATRIX WITH HIGH NOISY-TCP INSTANCES
The transition matrix is not identifiable by only exploiting noisy data without introducing additional assumptions, therefore we formulate the objective function for leaning the transition matrix based on the equation P( X) T (X X) as follow:
N
min T
Σ L (T (z.))
where f(x,) = Pv 1x))
(3)
N
n=1
5

---

Under review as a conference paper at ICLR 2023
To select the high-quality triplets (X, Y, Y) for the above objective, two conditions should be considered. First, it is necessary for f()) to out- put a precise clean class posterior, otherwise, T cannot be optimized in the correct direction, in
50000
Pseudo Clean nate
Selected Kumbar
Spwarmnn Correaciin Cowtfiient
40000
10000
0.8
the case is given and f(-) has been learned in advance and is fixed. As we discussed above,
20000
0.-
instances with high clean-TCP tend to have the correct pseudo label, and thereby precise clean class posterior, which satisfies this necessary condition. Second, the noisy-TCP should be high. By treating T (x) as whole predic- tor for y. the corresponding new objective is to predict ý. Therefore, high noisy-TCP instances naturally indicate the instance learned better and faster for predicting V. leading stable and fast learning.
10000
6.-
20
40 00 100
Epoch
Figure 4: Clean ratios of selected high noisy-TCP examples w.r.t. their pseudo labels with expo- nential growth of the selected number and Spear- man rank-order correlation coefficient between the noisy- and clean-TCP during our curriculum learn- ing on CIFAR 10 with IDN-0.4 for 100 epochs.
We discover that the noisy-TCP inherently has a strong correlation with clean-TCP so we can use select triplets fulfilling both conditions above. To see this, at each epoch, we calculate the Spearman rank-order correlation coefficient' between the noisy-TCP and clean-TCP of the whole dataset. Besides, we calculate the clean ratio of the selected high noisy-TCP instances w.r.t. their pseudo labels. In Figure 5, we show the data distribution in terms of clean- and noisy- TCP at epoch 50 (More data distributions at dif- ferent epochs are provided in Appendix C). The green regression line partially implies the linear correlation between the clean- and noIS Also, instances with correct pseudo labels are mainly distributed in the high TCP area, and vice versa. As shown in 4, the Spearman rank- order correlation coefficient above 0.6 after 10 epochs with consistent 0 p-value, roughly indicating that noisy-TCP strongly Spearman rank-order correlated with CCean-TCP for 100% sure. Meanwhile, the clean ratio is consistently above 0.8, which means those high noisy-TCP instances also have correct clean predictions and thereby probably precise clean class posterior. Note that the clean ratio decreases the late stage be- cause the curriculum selects almost all the data at the end. Overall, high noisy-TCP instances not only naturally stable for the objective predict ÿ but also satisfy the necessary condition have precise clean class posterior, which make them perfect examples for learning the transition matrix.
1.0
Glemn
077
Irstances with Correct Psoudo Labels
Whong Pscudo Labe's
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
3.1
Noksy TCP
Figure 5: Data distribution in terms of noisy- and clean-TCP at epoch 50 during our curriculum learn- ing on CIFARIO with IDN-0.4 for 100 epochs.
4.3 TCP GUIDED CURRICULUM LEARNING ALGORITHM
The main steps of our algorithm are summarized Algorithm with the complete procedure detailed in Appendix D. First, we warm up the feature extractor Φ, classification layer e by minimizing standard cross-entropy (CE) loss on noisy data, and meanwhile compute the clean-TCP for every instance. Then, we warm up the transition matrix layer í with high clean-TCP instances and obtain the noisy-TCP for every instance. From now on, iteratively, high clcan-TCP instances are fed to the clean classifier (green part in Figure 6) to train primary clean classifier with the clean CE loss, and based on the primary clean classifier, instances with high noisy-TCP are fed to the transition matrix (blue part in Figure 6) to train transition matrix with the noisy CE loss while the parameters of
'The Spearman rank-order correlation coefficient is nonparametric measure of the monotonicity of the relationship between (Zar, 2005).

---

Under review as a conference paper at ICLR 2023
Input data: samples with noisy labels (x.))
Feature Extractor (xx)
Classification Layer c((C0) P(ylx)
Clean CE Loss L_c(P(ylx),))
Cat
Dog
P(7x)) = (x)
Transition Matrix r(f(x)) > T(x)
Noisy CE Loss 1. x<PP()x))))
Figure 6: An overview of the proposed method. The second image of cat has noisy label as "dog". The transition matrix T()) = (()(.)) and classifier f(-) = c(o(-)) share common feature extractor.
Algorithm TCP Guided Curriculum Learning Algorithm.
Input: Noisy training sample D, feature extractor c, classification layer transition matrix layer t, f() c()()). T(-) = (()())), number sequences N₂ and N,, training epoch e.
1: Warmup clean- and noisy- TCP
2: Curriculum training:
3: for e in [1, (1....,,] do
4: Select [e]e] high noisy-TCP instances D,
5: Train & while fixing & and c on D, by minimizing Σn Cn (T (xn)) Un).
6: Select Ne[e] high dean-TCP instances De.
7: Train & and c on De by minimizing Σ Le (f(xn) (((nn,,0 Vn). where y" the pseudo label.
8: Fix and fine-tune and D by minimizing En= Ln (I (#n))((n) (zn). Un).
9: Record the clean and noisy prediction and calculate the clean- and noisy-TCP by Eq. equa-
tion 1.
10: end for
Output: Optimized feature extractor o, classification layer transition matrix layer I.
the primary clean classifier frozen. Then the clean classifier gets improved by being fine-tuned on the whole data with the fixed transition matrix. The clean- and noisy-TCP of every instance are updated at the end of each epoch. Finally, transition matrix with small estimation error and clean classifier with performance improvement can be obtained.
EXPERIMENTS
In this section, we examine how the proposed methods learn a robust classifier against instance- dependent noisy labels.
Dataset. We employ three widely used datasets, i.c., F-MNIST (Xiao et al., 2017), SVHN (Netzer al., 2011), and CIFARIO/I00 (Krizhevsky al., 2009), and four versions of the real-world noisy dataset CIFARION (Wei et al., 2022), and Clothing /M (Xiao et al., 2015). F-MNIST contains 60,000 training images and 10,000 test images with 10 classes. SVHN and CIFARIO both have 10 classes of images, but the former contains 73,257 training images and 26,032 test images, and the latter contains 50,000 training images and 10,000 test images while CIFAR 100 has 100 classes. CIFAR ION provides CIFAR10 10 images with human- annotated noisy labels obtained from Amazon Mechanical Turk. Four versions of CIFARION label sets are employed here, three of which are labeled by three independent workers (named above three sets (named CIFARION- -OW.NN Clothing
CATARION-1/221 and one of which is negatively aggregated from the
has 1M images with real-world noisy labels and additional 50k, 1 14k, 10k images with clean labels for training, validation and test, and we only use noisy training set in the training phase. For all the datasets, we leave out 10% of the training data validation set, which is for model selection. The final test model is selected with the highest validation accuracy.
Noisy labels generation. For clean datasets, artificially corrupt the class labels of training and validation sets following the eestance-depender noisy labels generalization method in Xia et al. (2020b). We generate noisy datasets of (0.1, 0.2, 0.3, 0.4, 0.5) five noise rates.
7

---

Under review as a conference paper at ICLR 2023
Table Means and stds of classification accuracy on CIFARIO with different label noise rates.
IDN-0.1
IDN-0.2
IDN-0.3
IDN-0.4
IDN-0.5
CE
74.49=0.29
68.2110.72
60.48=0.62
49.84=1.27
38.8612.71
Decoupling
74.09+00.78
70.01+0.66
63.05+0.65
44.27+1.911
38.63+22.32
MentorNet
74.65=00.6
70.56+0.34
65.42+0.79
46.22=#0.98
39.89+2.62
Co-teaching
76.99-0.17
72.99=0.45
67.22-1064
1..75=1.77
42.77=3.41
Co-teaching+
1..2
71.0740.77
64.77
47.73+2.32
39.47
Joint
76.89=0.37
73.89=0.34
69.03=0.79
54.75=5.98
44.72=7.72
DMI
75.02=00.45
69.8910.33
61.88=0.64
51.23#1.18
41.45=1.97
Forward
73,45=0:23
68.9940.62
60.21=0.75
47.17=2.96
40.75=2.09
Reweight
0..55
68.42
62.58+0046
50.12+0.96
41.08
T-Revision
0..69
69.32+0064
64.09+0037
50.38+0.87
42.57+3.27
CRUST
76.20=00900
1..41
71.06+2.21
64.59+3.32
52.50+0.81
CausalINL
79.08=0.40
1.065
67.70
49.19
1.58
PTD
78.71
75.02.10.73
71.46
56.15
49.07
Bayes
80.17=1332
19.51
7..93+1.99
69.53+3.24
57.42=4.37
TCP
$2.49-1..3
80.8
79.32 1.59
76.03
60.66
Baselines and measurements. On synthetic noisy datasets, without introducing data augmentation techniques and semi-supervised learning, we compare the proposed method TCP with the following baselines: (i). CE, which optimizes the standard cross-entropy loss noisy datasets. (ii). Decou- pling (Malach & Shalev-Shwartz 2017), which trains two networks on samples whose predictions different. (iii). MentorNet Jiang et al. (2018), Co-teaching (Han et al., 2018b), and Co-teaching+ (Yu et al., 2019), that mainly handle noisy labels by training on instances with small loss. (iv). Joint (Tanaka et al., 2018), which jointly optimizes labels and network parameters. (v). DMI (Xu et al., 2019), which uses novel information-theored
loss function to learn a robust classifier. (vi). Forward (Patrini et al., 2017), Reweight (Liu & Tao, 2016), and T-Revision (Xia al., 2019), that utilize class-dependent transition matrix T to correct the loss function. (vii). PTD (Xia et al., 2020b) and Bayes ( (Yang al., 2021), estimate deeendddn transition matrix under some additional assumptions; CRUST (Mirzasoleiman et al., 2020) iteratively selects subsets of clean data points that provide an approximately low-rank Jacobian matrix; CausalINI. (Yao et al., 2021) exploits the causal graph to estimate the transition relations between clean and noisy labels. On real-world noisy datasets, we apply the transition matrix learning and fine-tuning parts to the SoTA method Dividemix (Li et al., 2020), i.e., at each epoch, in addition to the Dividemix training, select high TCP data to learn the transition matrix and use it to fine-tune the whole data. Then compare this combined method TCP-D with the following SoTA methods: (i). PES (Bai et al., 2021). (ii) Dividemix (Li et al., 2020). (iii). CORES (Cheng et al., 2021). (iv). ELR+ (Liu et al., 2020). (v). JoCoR (Wei et al., 2020b). (vi). CAL (Zhu al., 2021). We
ResNet18 network for F-MNIST, a ResNet34 network for SVHN and CIFAR 10, ResNet50 network for CIFAR PreAct-ResNedt for CIFAR and a pre-trained ResNet50 network for Clothing IM. More training details are provided in Appendix E. Classification accuracy is employed to evaluate the performance of each model on the clean test set. Results over 5 trials on all datasets except Clothing IM, for which the result is over trial, are reported.
Comparison with the State-of-the-Arts. We compare TCP with multiple baselines using the same network architecture. Table 1 show the results CIFAR 10 with different rates of IDN from 0.1 to 0.5, respectively. TCP outperforms baselines across all datasets and noise rates. The improvement significant when the noise rate is large. Tables 5, 6, and 7 show the results on F-MNIST. SVHN, and CIFAR 100, which are provided in Appendix F. Table show the results on real-world noisy datasets CIFAR 1/233// and Clothing TCP-D consistently achieves the best test accuracy real-world noisy datasets. Note that the results of baselines on CIFARION are taken from the official leaderboard ht http: / www. noisy sybeelll..b........e com/.
Comparison on the transition matrix estimation error. We compare the transition matrix estima- tion error of method with the instance-independent method Forward (Patrini et al., 2017), and two
8

---

Under review as a conference paper at ICLR 2023
Table 2: Mcans and stds of classification accuracy on real-world noisy datasets.
CIFARION-1
CIFARION-2
CIFARION-3
CIFARION-W
Clothing 1M
PES (semi)
95.06=0.155
95.1944.23
95.22::.133
92.68=0.22
74.29
DivideMix
95.16=1.119
95.23
95.21
92.56=0.42
74.30
CORES
94.45+0.11
94.88+0.31
94.7441..03
91.66+0.009
73.24
ELR+
94.43+1044
94.20++.24
94.34+0.22
91.09+1.60
74.31
JoCoR
90.30:10.00
90.21110.19
90.111:0.21
83.3710.30
70.30
CAL
90.93+1031
90.75100.30
90.74.10.24
85.36:10.16
74.21
TCP-D
0.066
9..37
95.43.10.04
93.36.10.09
74.41
2.
Trward
Eayes
2.0
IDN-0.2
IDN-0.4
1.5
TCP
76.03 0.99
1.0
TCP w/o Do TCP w/o D.
0.89 79.08
2..28=2.26
74.98+1.64
05
Table 3: Ablation study results on CIFARIO.
Figure 7: Transition matrix estimation errors on CIFAR10 from IDN-0.1 to IDN-0.4.
instance-dependent methods PTD (Xia et al., 2020b) and Bayes (Yang et al., 2021). As shown in Fig- ure our method achieves the consistent best estimation error on CIFARIO with different noise rates.
Ablation study. We study the effect of removing different components of our methods to provide insights into what makes TCP successful in Table TCP w/o De indicates that we do not select high clean-TCP data D. to learn the clean classifier while TCP w/o D, indicates that we do not select high noisy-TCP data D, to learn the transition matrix and use it to fine-tune the clean classifier. Results show that the performances of both reduced methods decrease. Without De, the primary clean classifier cannot be learned, and thus the transition matrix cannot be learned well. Without D. the transition matrix is not learned, and thus the whole noisy data cannot be fully exploited to build the consistent classifier. To sum up, the learning of the clean classifier and the transition matrix benefit and boost each other.
CONCLUSIONS
In this paper, study the instance-dependent label noise (IDN) problem, which is more general and practical setting than the previously addressed instance-indeppndenl label noise problem. Targeting propose novel time-consistency metric, i.c., TCP for the IDN problem. Based on TCP, we can detect examples with clean labels or correct pseudo labels better than the existing measures, and allocate reliable triplets for learning the transition matrix. Then we design an assumption-free curriculum that learns the clean classifier, as well as the transition matrix simultaneously. Through extensive experiments, we empirically demonstrate that the proposed method remarkably surpasses the baselines on many datasets with both synthetic noise and real-world noise, and achieves the smallest transition matrix estimation error than cxisting methods.
the main challenges,

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. NeurIPS, 34, 2021.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pp. 41-48, 2009.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent lahel-noise learning possible. In ICMI., pp. 825-836. PMIR, 2021.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Support vector machines under adversarial label noise. In ACMI, 2011.
Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In ICCV, pp. 1431-1439, 2015.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: sample sieve approach. ICLR, 2021.
Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance-and label-dependent label noise. In ICML, 2020.
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In /CML., pp. 1311-1320. PMLR, 2017.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang. Dengke Dong, Matthew Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In ECCV, pp. 135-150, 2018.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A perspective of noisy supervision. In NeurlPS, pp. 5836-5846, 2018a.
Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018b.
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels robust. In ICML, pp. 4006-4016. PMLR, 2020a.
Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey label-noise representation learning: Past, present and future. arXiv preprint arXiv:2011.04406 2020b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training noisily labeled data with generalization guarantee. In ICLR, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data- driven curriculum for very deep neural networks on corrupted labels. ICML, pp. 2309-2318, 2018.
Tachyeon Kim, Jongwoo Ko, JinHwan Choi, Se- Se-Young Yun, et al. Fine samples for learning with noisy labels. NeurIPS, 34:24137-24149 2021.
Jan Kremer, Fei Sha, and Christian Igel. Robust active label correction. In AISTATS, pp. 308-316. 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
10

---

Under review as a conference paper at ICLR 2023
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels semi- supervised learning. ICLR, 2020.
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. ICML, 2021.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, pp. 1910-1918, 2017.
Sheng Liu. Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez- Granda. Early-learning regularization prevents memorization of noisy labels. NeurIPS, 33:20331-20342, 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. JEEE
Trunsactions pullern analysis and machine intelligence, 38(3)4477-41,, 2016.
Yang Liu and Hongyi Guo. Peer loss functions: rates. In ICML, 2020.
Learning from noisy labels without knowing noise
Xingjun Ma, Yisen Wang, Michael Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-drive learning with noisy labels. In ICMI., pp. 3361-3370, 2018.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In ICML, 2020.
Eran Malach and Shai Shwarzz Decoupling" when to update" from" how to update". In NeurlPS, pp. 960-970, 2017.
Naresh Manwani and P.S. Sastry, Noise tolerance under risk minimization. IEEE Transactions Cybernetics, 2013.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of neural networks against noisy labels. NeurlPS, 33, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurlPS, pp. 1196-1204, 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y.Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pp. 1944-1952, 2017.
Scott E Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR, 20155
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331-4340 2018.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS, pp. 838 846, 2015.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pp. 5552-5560, 2018.
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: simple and general method for sumi-super learning. In ACL, pp. 384-394, 2010.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In NewrIPS, pp. 5596-5605, 2017.
11

---

Under review as a conference paper at ICLR 2023
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, pp. 239-847, 2017.
Xiaobo Wang. Shuo Wang, Jun Wang. Hailin Shi, and Tao Mei. Co-mining: Deep face recognition with noisy labels. In ICCV, pp. 9358-9367, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: joint training method with co-regularization. In CVPR, June 2020a.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, pp. 13726-13735, 2020b.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu, Learning with noisy labels revisited: A study using real-world human annotations. In ICLR, 2022.
Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. In NeurlPS, 2020.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurlPS, pp. 6835-6846, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106.000 00445, 2020a.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurIPS, 2020b.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXhn:1708.07747 2017.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. d__mi: An information-theorrticc noise-robust loss function. In NeurIPS, 2019.
Shuo Yang, Erkun Yang. Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent label-noise transition matrix using dnns. arXiv preprint arXiv:2105.13000 2021.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under a structural causal model. NeurlPS, 2021.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement benefit co-teaching? In ICML, 2019.
Xiyu Yu, Tongliang Liu, Mingming Gong, Kayhan Batmanghelich, and Dacheng Tao. An efficient and provable approach for mixture proportion estimation using linear independence assumption. In CVPR, pp. 4480-4489, 2018.
Jerrold H Zar. Spearman rank correlation. Encyclopedia of biostatistics, 7, 2005.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018
Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. ICML, 2021.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, pp. 8778-8788, 2018.
12

---

Under review as a conference paper at ICLR 2023
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In ICML, pp. 11447-11457, 2020.
Tianyi Zhou, Shengjie Wang, and Jeffrey Bilmes. Curriculum learning by dynamic instance hardness. NeurIPS, 33:8602-8613,, 2020.
Tianyi Zhou, Shengjie Wang. and Jeff Bilmes. Robust curriculum learning: from clean label detection to noisy label self-correction. In ICLR, 2021.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In CVPR, pp. 10113-10123, 2021.
13

---

Under review as a conference paper at ICLR 2023
A MORE EMPIRICAL STUDY REGARDING FIGURES AND 2
A.1 MORE EMPIRICAL STUDY REGARDING FIGURE.
Figures 8 and 9 show the results
practical dataset CIFARI00 and IDN noise 0.4 higher IDN noise 0.6 with different partitions (2:6:2), which demonstrate that our claims hold in general and do not change sensitively with the partition ratios.
Figures 10 and 11 show the results on datasets CIFAR100 and SVHN with different model archi- tectures. Conclusions from Section 3 are based the memorization effect of overparameterizee DNNs. Therefore, they hold better for deeper DNNs (ResNet50) than the shallower DNNs (AlexNet). Moreover, for AlexNet on SVHN, the high TCP group partitioned an early stage has overlap with the middle TCP group. Overall, the results demonstrate that our conclusions in the paper holds true and generalize to other architectures and datasets.
Regr
Redion Sey
03
"CP
15
04
2
41
044
02
1pac
Ipoc
(a) Partitioned at start stage.
(b) Partitioned at early stage.
(c) Partitioned at an end stage.
bwe for Crons
0A
2
34
04
44
93
20
300
08
303
(d) Partitioned at start stage.
(e) Partitioned at a early stage.
(1) Partitioned at an end stage.
Figure& 8: TCP (mean and std.) of three groups partitioned by the TCP calculated the start stage (epoch 5), early stage (epoch 30). and end stage (epoch 95) during training a ResNet50 on CIFAR 100 with IDN-0.4 for 100 epochs. The first row is partitioned by high TCP (10%), middle TCP (80%). and low TCP (10%). The second row is partitioned by high TCP (20%), middle TCP (60%), and low TCP (20%).
A.2 COMPARISON WITH SOTA SELECTION METHODS
Following the same setting as Figure 2, we select 5,000 confident examples at every epoch according to six types of selection criterion, i.e., instantaneous prediction InP, (zr), instantaneous loss f(x). time-consistency of prediction TCP, (3r), time-consistency of loss, and two
0.9
08
0.6
constriency Prediction
0.5
lime-cursistenryy
SOTA confident sample selection methods: FINE (Kim et al., 2021) and Topological Fil- ter (Wu et al., 2020). Then we count the number of instances with clean labels and calculate the clean ratios. As shown in Figure 12, at the starting stage, when the model just learns the clean data while has not fit the noisy data, FINE and Topological Filter perform perform well. As the training goes and the model fits the noisy data, our method achieves the best selection clean ratio.
0.4
Inctantarecu Frediction
Iratantaracia Loss
Random Selection
0.2
FINE
Topolog cal Fiber
20
40
60
80
100
Spoch
Figure 12: Clean ratios of the selected top 5000 instances ranked by six kinds of instance hardness measures, respectively. The clean ratio of randomly selected instances is 0.6 since the noise rate is 0.4.
14

---

Under review as a conference paper at ICLR 2023
Hon T2 Orcp
170
43
02
A0
23
20
80
325
04
30
1036
(a) Partitioned at start stage.
(b) Partitioned at early stage.
(c) Partitioned at an end stage.
1.1
1A
17 Tr
08
(d) Partitioned at start stage.
(c) Partitioned at early stage.
(1) Partitioned end stage.
Figure 9: TCP (mean and std.) of three groups partitioned by the TCP calculated at the start stage (epoch 5), early stage (epoch 30), and end stage (epoch 95) during training a ResNet50 on CIFAR 100 with IDN-0.6 for 100 epochs. The first row is partitioned by high TCP (10%), middle TCP (80%), and low TCP (10%). The second row is partitioned by high TCP (20%), middle TCP (60%), and low TCP (20%).
3.4
Button Seat
3.5
T7
0.4
40
04
04
Pyuce
Pyuce
(a) Partitioned at start stage.
(b) Partitioned at early stage.
(c) Partitioned at an end stage.
Figure 10: TCP (mean and std.) of three groups (high TCP (10%), middle TCP (80%), and low TCP (10%)) partitioned by the TCP calculated at the start stage (epoch 5), early stage (epoch 30), and end stage (epoch 95) during training a ResNet50 on CIFAR100 with IDN-0.4 for 100 epochs.
H6gh 12 099
03
03
0
63
13
122
08 20
4020
(a) Partitioned at start stage.
(b) Partitioned at carly stage.
(c) Partitioned at an end stage.
Figure 11: TCP (mean and std.) of three groups (high TCP (10%), middle TCP (80%), and low TCP (10%)) partitioned by the TCP calculated at the start stage (epoch 5), early stage (epoch 30), and end stage (epoch 95) during training AlexNet on SVHN with IDN-0.4 for 100 epochs.
15

---

Under review as a conference paper at ICLR 2023
B THEORETICAL ANALYSIS ON THE EFFECT OF INTRODUCING INSTANCES WITH PSEUDO LABELS
Consider the situation we have labeled set L (in practice it can be the selected confident examples set) and one unlabeled instance x'. By training on for one step, we have
B++1 0g + Σ
Vol (z; 0e)
sEC
and by training on L and r' for one step. we have
0 01 + "
Σ LEEC
Vol (2; Vol
where Or denotes the network parameters step t and denotes the learning rate.
The Taylor expansion loss & (x;; 8) at the point Oo is:
Σ (z: + Vol (2: (0 00) + o +(0(--)))) LEEC
gou (0)
Then evaluate the forgetting effect of introducing instances x' with its pseudo label to the training set by checking the change of loss over the labeled set L with x' added or not. If adding x' does not cause vital change, we can conclude that does not lead to catastrophic forgetting of the learned examples with correct labels. Therefore, we calculate the change of loss over the labeled set by
(x;) - (sx;042)
1
goe (0++2)-
-gr (0++1)
Vol (z':0') Σ Vol (2;00)
WEC
0E (x': 0*) age
00 OE (2(;0'))
Ōt
Ot
de (z':0')
0p. (x')
Opt (x')
Ot
where Pt p. (x') is the probability of z' belonging to V at step f, and i/ is the prediction (pseudo label) of x' at step f. The second line holds because we omit the second and higher order terms the Taylor expansion in Eq equation 4. Then, with cross-entropy loss employed, we have
a8 (2':0'))
alog(p (x'))
Opp (x')
Op. (x')
86 (x')
Pt
Next, by using (p:+ (x') - 2 (a') to approximate
0x.
(a')
we have
Gt
1
Σ ZEC
é (z; (11)) - xx; o )]
20
(5)
Pt (xc)
06 (xx) is very close to Pr (x') because it has been verified Since x' is selected with high clean-TCP, P1 in Figure that instances with high clean- in the early stage maintain their high clean-TCP the future, which means the loss change be bounded with very small value. Therefore, exploiting high clean-TCP instances with pscudo labels helps to correct corrupted labels and learn a clean classifier without causing catastrophic forgetting of the learned examples with correct labels.
16

---

Under review as a conference paper at ICLR 2023
C DATA DISTRIBUTION IN TERMS OF CLEAN- AND NOISY-TCP AT DIFFERENT TRAINING STAGES
1.05
1.00
0.55
Clenn
0.85
0.80
0.75
Iestarres with Connert Peradin abels
Inabences with Fornect Promctt watc Irsilinces with Wiong Pacudd Labds
Iistaries with Whing Pacudo Labels
0.70
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
3.0
Koisy TOP
Noley TCP
(a) Epoch 10.
(b) Epoch 30.
2.00
0.95
Ceenn
0.90
0.80
Instances with Correct Pocudo Labels Instences Whing Pinudo Labels
Instances with Correct Prei, do Labels Inatances Veong Paeude Labels
0.70
0.6
0.7
6.8
0.9
1.0
0.6
0.7
0.8
0.5
1.0
Koisy TCC
Nokyy "CP
(c) Epoch 80.
(d) Epoch 100.
Figure 13: Clean ratios of selected high noisy-TCP examples w.r.t. their pseudo labels with expo- nential growth of the selected number and Spearman rank-order correlation coefficient between the noisy- and clean-TCP during our curriculum learning on CIFARIO with IDN-0.4 for 100 epochs.
D ALGORITHM
E TRAINING DETAILS
We use ResNet18 network for F-MNIST, ResNet34 network for SVHN and CIFAR10, ResNer50 network for CIFAR 100, PreAct-ResNet18 for CIFARION, and a pre-trained ResNet50 network for ClothingIM The transition matrix is modeled by linear layer which transforms the latent representation vector to a c² vector, and then reshaped to a x c matrix. The batchsize set to 32 for Clothing IM and 128 for others. The weight decay is set to 1e 54 0, 5e 4 and 0.001 for F-MNIST, SVHN, CIFAR10/100, CIFARION, and Clothing IM, respectively. For synthetic noisy datasets, the Adam optimizer. At the warmup clean-TCP stage, the learning rate is initialized
17

---

Under review as a conference paper at ICLR 2023
Algorithm 2 TCP Guided Curriculum Learning Algorithm.
Input: Noisy training sample D, feature extractor o, classification layer c, transition matrix layer f()) = c(o(-)), T()) = t(((()),, number sequences Ne and N,, training epoch C2 and eg.
1: Warmup clean-TCP:
2: for in (......,.. { do
3: Train & and on D by minimizing standard CE loss Σn= & (f(z.),
4: Record the clean prediction and calculate the clean-TCP.
5: end for
6: Warmup noisy-TCP:
7: for in {1, {1.... E2) do
8: Select No[e] high clean- instances as De.
9: Train c, and on De by minimizing Em=1 Ln (f(=x) (n))).))+t.) <(T ((TT(z,)) (zn)f(zn) Fn).
Record the clean and noisy prediction and calculate the clean- and noisy-TCP.
11: end for
Curriculum training:
13: for in {1...., E3) do
14: Select Ne(e2 + e| high noisy-TCP instances as D.,
15: Train t while fixing Φ and on D, by minimizing E Cn (T (In)f(In), Dn).
16: Select Ne(e2 + e) high clean-TCP instances as De.
17: Train & and
D. by minimizing Σ La (J(nm, Ii.) where y² is the pseudo label.
Fix and fine-tune and on D by minimizing Σ Ln (T (#n) Un). Record the clean and noisy prediction and calculate the clean- and noisy-TCP by Eq. equa-
19:
tion 1.
20:
end for
Output: Optimized feature extractor o, classification layer c, transition matrix layer t.
to 0.001 and decayed every 5 epochs with 50 epochs in total by a factor of 0.1, 1/3, and for F-MNIST, SVHN, and CIFAR10/100 respectively. the rest of 100-epoch training, the leaning rate of the feature extractor and classification layer is 1e_4 and divided by 10 at epoch 30 and 80; the leaning rate of the transition matrix layer t 3e-4 before epoch 30 and 1 otherwise. The learning rate for fine-tuning is le-6, For real-world noisy dataset CIFARION and Clothing we follow the optimization method as Dividemix. For CIFARION ION, the warmup clean-TCP stage, the learning rate is initialized to 0.001 and decayed every 5 epochs with 50 epochs in total by a factor of 1/3. the rest of 300-epoch training, the leaning rate the transition matrix layer is 6e before epoch 80 and 2 between epoch 80 and 150, and 2 otherwise. The learning rate for fine-tuning 2e before epoch 80, and -4 between epoch 80 and 150, and 2e-4 between epoch 150 and 250, and -5 otherwise. For Clothing IM, in the warmup clean-TCP TCP stage, the learning rate is initialized to 0.002 and decayed every 2 epochs with 5 epochs in total by a factor of 1/3. In the rest of 20-epoch training, the leaning rate of the transition matrix layer is 6e -4 before epoch 8 and -5 between epoch 8 and 12, and 5e otherwise. The learning rate for fine-tuning is 2e 4 before epoch 10, and 6e between epoch 10 and 14, and 2e between epoch 14 and 17, and 2e otherwise.
E. TRAINING TIME
Due to the light-weight and simple network architecture, our method is more scalable than those methods, which employs dual networks or requires data augmentations for semi-supervised learning. We report the time costs below to demonstrate this advantage.
and
Table The average time of training each component on CIFARIO and CIFARIOS with ResNet34 on NVIDIA 3090.
Standard training
Fine-tune
Estimating
CIFARIO CIFAR100
38.42s 37.37s
46.19s 47.06s
1.34s
1.57s
18

---

IDN-0.5 Under review as a conference paper at ICLR 2023
The additional cost caused by estimating T and fine-tune is small. For each epoch, the additional time cost estimating- part is neglectable when compared with the cost of one standard training epoch minimizing the cross-entropy loss. This is because estimating only updates the parameters of a e x linear layer, where e the number of classes. The time cost of fine-tune part is slightly bigger than standard training epoch. Fortunately, both parts not necessary be applied in every epoch. In our experiments, we only apply them at the last 50 epochs. Moreover, since traning clean classifier part (line 7 in Algorithm and estimating part only involves the high clean-TCP and high noisy-TCP data rather than the whole data, which save plenty of time for the fine-tune part. Therefore, in practice, our method can easily adapt and scale meet the realistic settings.
EXPERIMENT RESULTS
Table 5: Means and stds classification accuracy on F-MNIST with different label noise rates.
IDN-0.1
IDN-0.2
IDN-0.3
IDN-0.4
CE
88.54=0.31
88.38=0.42
84.22=0.35
68.8620.78
51.4210.66
Decoupling
89.27=0.31
86.5010.35
85.33=0.47
78.5420.53
57.32-2.11
MentorNet
90.00=0.334
87.02=0.411
86.02=0.82
80.12=0.76
58.62+1.36
Co-teaching
90.82+0.33
87.89=0.41
86.88=0.32
82.78=00.55
63.22+1.58
Co-teaching+
90.92.1.551
89.7740.45
88.52:10.45
83.57+1.77
59.32
Joint
70.94
56.83+10.45
51.27
44.24
30.45
DMI
91.98=0.62
90.3310.21
84.81=0.44
69.01=1.87
51.64
Forward
89.05=0.43
88.61
84.27-0.46
70.25=1.28
57.33
Reweight
90.33++.27
89.70+00.5
87.04+0.35
80.29+0.89
63.27+1.33
T-Revision
91.56++0311
90.68+0.66
89.46=1045
84.01+1.24
68.99+1.04
CRUST
89.53
89.20=
8..98
83.48=1.55
69.59+3.60
CausalINL
90.14+0.31
88.83+0.37
15.38+1.49
83.82+2.29
69.55+4.11
PTD
91.0111.22
90.03+0..2
87.68
84.03+0.52
1.74
Bayes
92.0111.22
91.42
89.64-1.41
81.21
2.46
TCP
92.64 0.22
92.15 0.38
91.62 0.59
90.56 1.799
77.49 1.499-2200 2.88
Table 6: Means and stds of classification accuracy SVHN with different label noise rates.
IDN-0.1
IDN-0.2
IDN-0.3
IDN-0.4
IDN-0.5
CE
90.77
90.23+10.22
86.33
65.66
48.01 +4.59
Decoupling
10.49
90.47+4066
85.27+0.34
82.57+1.45
4..56
MentorNet
90.28=0.12
90.3740.37
86.49=0.49
83.7510.75
40.2743.14
Co-teaching
91.33=0.31
90.5610.67
88.9320.78
85,47=0.64
45.90-2.31
Co-teaching+
93.05+1.20
91.05+0.82
85.33+2.71
57.2443.77
42.56=3..5
Joint
86.01+0.34
78.58+0.72
76.34+0.56
65.14+1.72
46.78+3.77
DMI
93.51+1.09
93.22+0.62
91.78+1.54
69.3442.45
48.93+2.34
Forward
90.89-10.3
90.65+1..2
87.3210.59
78.46:2.58
46.27+3.90
Reweight
92.49-10.44
91.09-10.34
90.25
84.4
45.46-13.56
T-Revision
94.2410.53
94.00+0.88
93.01+0.83
88.63+1.37
49.02+4.33
CRUST
93.22+1.32
91.55-10.36
88.64+1.43
80.75+2.78
58.3012.77
CausalINL
92.38
91.4010.86
90.23+1.60
84.50=1.71
68.0645.12
PTD
93.2110.45
92.36-0.66
90.57=0.42
86.78-0.63
55.8813.73
Bayes
94.7140.44
94.02+1.32
91.38+1.94
85.55+3.17
75.46+3.79
TCP
9.11
94.60+10.20
93.92+11.77
94.09-10.34
8.440
19

---

Under review as a conference paper at ICLR 2023
Table 7: Means and stds of classification accuracy CIFARI00 with different label noise rates. Note that PTD is not applicable to CIFAR 100 which has large classes due to its matrix factorization component.
IDN-0.1
IDN-0.2
IDN-0.3
IDN-0.4
IDN-0.5
CE
36.80
31.64
30.67+2.67
24.00=1..76
20.2411.49
Decoupling
37.16=0.86
33.01
31.65+2.62
24.72+2.51
20.13+2272
MentorNet
37.95
33.72#1.03
32.04=1.97
26.93=2.35
21.86#2.30
Co-teaching
38.57=0.95
35.60+1.49
33.77=1.91
26.17=2.35
21.96=2.51
Co-teaching+
37.92+1.04
34.51+1.43
33.13:2.04
25.98
2..48
Joint
38.96-10773
35.91+1.22
34.23+1.47
28.7543.69
23.89
DMI
37.60-0.84
34.72+1.38
32.87=1.60
28.60
23.25+2.81
Forward
37.00=1.555
32.72+2.67
31.6012.84
27.2442.89
21.13
Reweight
37.11=0.98
33.98+1.68
32.60+1.22
27.83+1.27
22.01+3.26
T-Revision
38.03+11005
34.42+2.32
33.60
28.1543.69
22.12+3.67
CRUST
1..9
41.75+11.2
38.60=2.01
32.42+5.23
24.41+2.12
CausalINL
38.02=0.78
36.31
32,23+9.23
27.63+4.38
22.42+2.16
Bayes
40.76
36.56+11.20
29.26
24.38
17.66
TCP
49.65 0.43
46.28 2.56
44.12 1.92
39.88 0.62
29.45 2.35
20


Reviews:
[{"review_id": "ysS-1KanTja", "reviewer": "Reviewer_FeeE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies robust learning on instance-dependent label noise. It proposes to discover the noise transition matrix and identify clean labels at the same time during training. It also proposes a metric to measure the probability of a label being correct, which is based on historical predictions of an instance during training. Such a metric can be used to better identify both correct labels and noisy labels, which facilitates the learning of noise transition matrix and clean labels.", "summary_of_the_review": "I believe there are some interesting ideas in this paper. But right now, the clarity issue makes it almost impossible to judge the novelty and contribution. I think significant revision is required to improve the paper.", "strength_and_weaknesses": "Strength:\n* The paper is overall organized well\n* Introduction to the related work is thorough and informative\n* Learning the transition matrix and clean classifier simultaneously seems to be an interesting idea\n\nWeaknesses:\n\n\n**Clarity:** I believe overall this paper is not well-clarified, which greatly hinders reading and comprehension. I'll list some of the clarity issues below for reference.\n* First, the background section is not sufficient in introducing the terms. For example, terms like \"pseudo label\", \"catastrophic forgetting\", \"corrupted labels\" in section 4.1 are never defined. I am having a hard time understanding this section without a clear definition of these terms. Most of the time I have to guess their meanings based on my own background knowledge.\n* What does Equation (2) mean? I guess the point here is to prove the change of loss is small for high-TCP examples, but why this is important? I think the proposed metric is not based on losses. \n* Quote, \"instances with clean labels are mutually consistent with each other in producing gradient updates\". This sentence is frequently mentioned in the paper, e.g. Section 3, 4.1, and thus I believe it should be important. But what does this mean? I am expecting some formula to clarify this but there is no. I might miss something here.\n* Figure 2 shows the proposed metric is good at identifying correct labels when there is label noise. But how does this label noise generated? The authors only mentioned \"manually add IDN at 0.4 noise rate (IDN-0.4) onto a benchmark dataset CIFAR10\", but I didn't find any details of this, even in the appendix. I think such detail is critical here, as we would expect the proposed metric not to specialize on some particular type of label noise. Experiments on most label noise settings here can also be helpful but I didn't find any figures similar to Figure 2, but experimenting on different label noise settings.\n\n**Quality**: There are quite a few unjustified premises in this paper, which cause difficulty for one to judge the contribution. For example, \n* In section 3, quote, \"Apparently, at the instance level, the one-hot prediction of an instance is a more robust metric than the loss because the former has a tolerance to the change of predicted class posterior while the latter has not\". But this doesn't appear apparent to me. I can also justify an opposite claim in a similar way, namely the loss considering the entire probabilistic prediction thus is more robust.\n* In Section 4.2, quote, \"high noisy-TCP instances naturally indicate the instance is learned better and faster for predicting y, leading to stable and fast learning.\". But I cannot see the reason here. Why is this premise natural?\n* I also have some questions about the experiment results. In Table 1, why baselines such as Dividemix are missing? Is it because Dividemix is not applicable here? In Table 2, why a new method is introduced (TCP-D)? What about the performance of the original method? And there are very few descriptions of this new method.\n\n**Novelty**:\n* I believe the proposed metric itself is not novel. Using prediction stability to identify correct or noisy labels is seen in previous works, e.g. [1].\n* The idea that using a designated model to learn the transition matrix is relatively novel to me. But apart from one sentence in the abstract, I didn't find any further explanation or justification for this idea.\n\n[1] An Empirical Study Of Example Forgetting During Deep Neural Network Learning. Toneva et al.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oJ4WrnDOU6M", "reviewer": "Reviewer_vCae", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work targets to the instance-dependent noisy-label learning problem and proposes a new curriculum learning-based algorithm. In particular, the authors calculate the 'time-consistency of prediction' (TCP) scores that indicate how consistent are the predictions of samples over the course of training. Then, the proposed framework selects anchor instances by the TCP scores and estimates the transition matrices. Experiments show the proposed method achieves better performance than the baselines. ", "summary_of_the_review": "The originality is doubtful and the empirical study is weak. So I vote for rejection.", "strength_and_weaknesses": "Pros:\n\n1. This work provides a different perspective on selecting clean examples for noisy-label learning by the consistency of predictions, which brings some insights to the community. \n\n2. The authors provided thorough analytical results of the TCP scores.\n\nCons:\n\n1. My first concern is that the proposed TCP score, while seemingly new, may be analogous to the pseudo-labeling procedure of semi-supervised learning (SSL) and inherently not a new method. In SSL, methods like FixMatch clip the model prediction on unlabeled data as their pseudo-labels for training; Pi-Model/Mean-Teacher adopt a time-consistent prediction as pseudo-labels. When the model is confident to one instance (for example, max prediction > 0.95), its prediction would also be stable as well and has TCP scores. Such a pseudo-labeling method had also adopted in the noisy-label learning regime [1]. In fact, I doubt TCP is inherently a semi-supervised learning algorithm and may not be better than such a pseudo-labeling strategy. \n\n2. The relationship between the TCP score and the instance-dependent noise is unclear. Why is TCP particularly important to the ID setup? In effect, as the proposed method also integrates the SSL techniques, I believe simple pseudo-labeling can also accomplish the anchor sample selection procedure for transition matrix estimation. \n\n3. The empirical study is weak for me. \n\n    3.1 First, the real-world noisy label CIFAR dataset has a version of CIFAR-100-N, but the results are reported in neither Table 2 nor Appendix. Does the proposed method underperform on this dataset? \n\n    3.2 Some important baselines are missing. For example, the UNICON, and SOP methods have shown promising results. The seminal DivideMix method is not compared in Table 1. While it's not mainly designed for ID NLL, I believe it can also handle this task empirically. Notably, TCP might be a semi-supervised NLL method and simply comparing it with naive NLL baselines is unfair.\n\n    3.3 Second, the ablation study is really weak. The authors should thoroughly analyze the effectiveness of different components. Does the estimated transition matrix really work? What if we directly run a dividemix model with TCP-selected samples? Can we replace the TCP-based selection with a pseudo-label-based selection for the transition matrix? \n\n    3.4 The ablation is mainly conducted on CIFAR-10, which is not convincing. \n\n4. I didn't find the definition of \\hat{y}, which made me hard to understand Eq. (1). \n\n[1] Li J, Xiong C, Hoi S. MoPro: Webly Supervised Learning with Momentum Prototypes[C]//International Conference on Learning Representations. 2020.\n\n[2] Karim N, Rizve M N, Rahnavard N, et al. UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 9676-9686.\n\n[3] Liu S, Zhu Z, Qu Q, et al. Robust Training under Label Noise by Over-parameterization[J]. ICML, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ffuaZTNFits", "reviewer": "Reviewer_GBT4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method for training a classifier on data with Instance-Dependent Noisy Labels (IDN Labels, i.e. where label noise is a function of the instances themselves).  Their technique is within a class of techniques that monitor a heuristic during training that measures some notion of noise in labels, and then using instances selected as \"clean\" and \"noisy\" then learn a transformation matrix that describes the relationship between true and noisy labels in an instance-dependent manner.  The main novel observation is the the heuristics in prior work are sensitive enough that monitoring them from one training epoch to another doesn't provide clear signal  as to which instances and \"clean\".  They instead use a TCP metric that is a running average of whether the predicted label has changed between gradient updates.  They show that if you assume that their metric is relatively stable, then their curriculum learning approach that introduces new instances during training does not result in catastrophic forgetting.  Empirically, the proposed method outperforms a number of baselines, both with real and synthetic noise.", "summary_of_the_review": "While the clarity of the paper is a barrier for understanding and the proposed technique is not of significant novelty, the authors provide strong intuition for their approach and follow that with strong empirical performance.  Because this technique seems to be the empirical leader for the IDN Labels learning problem, I am leaning towards acceptance.  I do think for a camera ready copy, the authors should take a fine-grained critical look at the writing and organization of the paper to ensure that the main contributions are more clearly articulated.", "strength_and_weaknesses": "Strengths\n1. Clearly techniques that learn from instance-dependent noise have considerable practical applications.\n2. Their approach is relatively simple and seems like it could be implemented easily with common machine learning packages.\n3. In terms of classification accuracy, the proposed approach achieves impressive performance relative to baselines.\n\nWeaknesses\n1. The paper suffers from poor organization which makes its somewhat difficult to follow.  Namely, there needs to be a self-contained \"preliminaries\" before describing the techniques that are introduced.  As it is written right now, lots of notation and leveraged prior work are introduced throughout the paper.  As a result, it is both difficult to follow the notation and difficult to identify what specifically was novel about the proposed techniques.  I recommend taking a step back and thinking about how to reorganize the paper entirely, and also consider discussing the most related IDN label learning algorithms in a more formal way to establish them as a point of reference to the proposed new contributions made by this work.\n2. The claim of the proposed technique being immune to catastrophic forgetting is discussed as a formal contribution, but it neither formally defined nor strongly argued.  There needs to be a formal statement of what is being proved.  Also, the proof hinges on the statement: \"Since x′ is selected with high clean-TCP... has been verified in Figure 1\".  This is not a formally proved statement, which makes the \"proof\" more of an empirical observation.\n3. The curriculum learning algorithm requires training two models to completion before training begins.  It is not clear how computationally expensive this is relative to the baselines.  To provide a more fair understanding of the performance of the proposed technique, I think wall run times should be provided for the training of the models in the empirical results section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3d08JaLz0mH", "reviewer": "Reviewer_zfb4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel metric, Time Consistency Prediction (TCP), as criteria to guide curriculum learning for jointly training the label transition matrix and clean classifier.", "summary_of_the_review": "I believe the novelty and contribution of the paper outweigh the weaknesses of current theories, results and writing. I'm happy to increase my score if the authors can address some of my concerns.", "strength_and_weaknesses": "Strength:\n\n1. The proposed metric TCP for curriculum learning is theoretically motivated. In particular, empirical results show that it is a very interesting alternative that outperforms several existing loss based selection criteria.\n2. Integrating the TCP, Algorithm 1 is a nice general framework to combine both curriculum learning and label noise transition learning. Empirical experiments are quite strong.\n\nWeaknesses / Major Concerns:\n\n1. Regarding the motivation of TCP, in particular TCP vs. time consistency of loss (TCL), I find it a bit surprising that TCL works so much worse than TCP (Fig 2). The authors explained that 1-hot prediction is more robust than loss at instance level (second paragraph of Sec 3), but the authors later also said clean-TCP instances' loss change can be bounded with a very small value (left to Fig 3). May the authors give more justifications?\n2. When motivating the curriculum learning in Sec 4, Fig 3 sees a drastically decreasing clean ratio of original noisy labels, while Fig 2 sees TCP clean ratio still stay high towards the end of training. Why is this? Is this due to the linearly increasing curriculum size in Fig 3? There is some prior work that shows fixing the number of instances being selected given small losses is probably not a good idea (e.g. https://arxiv.org/pdf/2104.02570.pdf), should consider setting a (dynamic) loss threshold instead.\n3. Continuing on the previous question, how is the number of selected instances decided at each epoch in the experiments (e.g. linear/exponential growth as done in Fig 3/4)? Is there a way to dynamically control the number of selected instances with TCP?\n4. Despite that the synthetic data experiments look very promising, the authors decide to use a variant of TCP-D in the real world data experiments (Tab 2). TCP-D extends DivideMix by selecting high noisy-TCP data to learn the transition matrix and using it to fine-tune the whole data. Given my understanding of the contribution of the paper, would it make more sense to actually replace the GMM-based sampling selection part of DivideMix by TCP guided curriculum? Why the particular choice of TCP-D? Did the authors try other extensions of DivideMix inspired by TCP?\n\nMinor comments:\n\n1. Should the authors include EMA over loss (Zhou et al. 2020) as it has been shown to be more robust than instantaneous loss, since TCP also considers the entire training history?\n2. I think TCP w/o D_t in Tab 3 performs well enough to be standalone methods given that they are much simpler alternatives to the full model without the whole label transition or extra fine tuning on entire data (pending more results?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject