Class-Dependent Label-Noise Learning with Cycle-Consistency Regularization
Authors: De Cheng
Yixiong Ning
Nannan Wang
Xinbo Gao
Heng Yang
Yuxuan Du
Bo Han
Tongliang Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Label-Noise Learning
Transition Matrix

Abstract: In label-noise learning, estimating the transition matrix plays an important role in building statistically consistent classifier. Current state-of-the-art consistent estimator for the transition matrix has been developed under the newly proposed sufficiently scattered assumption, through incorporating the minimum volume constraint of the transition matrix T into label-noise learning. To compute the volume of  T, it heavily relies on the estimated  noisy class posterior. However, the estimation error of the noisy class posterior could usually be large as deep learning methods tend to easily overfit the noisy labels. Then, directly minimizing the volume of such obtained T could lead the transition matrix to be poorly estimated.  Therefore, how to reduce the side-effects of the inaccurate noisy class posterior has become the bottleneck of such method. In this paper, we creatively propose to estimate the transition matrix under the forward-backward cycle-consistency regularization, of which we have greatly reduced the dependency of estimating the transition matrix T on the noisy class posterior. We show that the cycle-consistency regularization helps to minimize the volume of the transition matrix T indirectly without exploiting the estimated noisy class posterior, which could further encourage the estimated transition matrix T to converge to its optimal solution. Extensive experimental results consistently justify the effectiveness of the proposed method, on reducing the estimation error of the transition matrix and greatly boosting the classification performance.

Class-Dependent Label-Noise Learning with Cycle-Consistency Regularization
De Cheng'; Yixiong Ning's T Namnan Wang
Xinbo Gao², Heng Yang', Yuxuan Du Bo Han', Tongliang Liu Xidian University, Chongqing University of Posts and Telecommunication,
Shenzhen AiMall Tech. Co., Ltd., Explore Academy, Hong Kong Baptist University, TML Lab, The University of Sydney.
Abstract
In label-noise learning, estimating the transition matrix plays important role in building statistically consistent classifier. Current state-of-the-art consistent estimator for the transition matrix has been developed under the newly proposed sufficiently scattered assumption, through incorporating the minimum volume constraint of the transition matrix T into label-noise learning. To compute the volume of T, it heavily relies on the estimated noisy class posterior. However, the estimation error of the noisy class posterior could usually be large deep learning methods tend to easily overfit the noisy labels. Then, directly minimizing the volume such obtained T could lead the transition matrix be poorly estimated. How to reduce the side-effects of the inaccurate noisy class posterior remains unsolved. this paper, we creatively propose to estimate the transition matrix under forward-backward cycle- cole--oosistency regularization, of which we have greatly reduced the dependency of estimating the transition matrix T on the noisy class posterior. Extensive experimental results consistently justify the effectiveness the proposed method, on reducing the estimation error the transition matrix and greatly boosting the classification performance.
Introduction
Deep learning based algorithms rely heavily large-scale annotated training data. However, it is
often extremely costly and sometimes even infeasible to accurately annotate such massive dataset [3]
An usual way to address this issue is to collect large-scale training data using cheap methods, e.g., from the crowd-soureing platform (36) or online query search engine 3 which could inevitably yield label noise. Training deep model with label noise significantly degrade the performance [13 28
Therefore, how to mitigate the side-effects of the label noise automatically becomes very important, and it has drawn increasing attention recently.
Recent studies show that estimating the transition matrix plays an important role in building sta- tistically consistent classifiers for label-noise learning, since the transition matrix can well model the noise generation process. These methods guarantee that classifier learned from the noisy data could approach to the optimal classifier defined the clean risk asymptotically the size of the noisy training data increases [33. The basic principle is that the clean class posterior P(Y = x be inferred by the noisy class posterior P(Y X and the transition matrix T(x), where
"Equation contribution.
Corresponding author. 'We define P(Y]X = x) [P(Y = 1|X - x)...., P(Y C|X = X where C' represents the number
of classes.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Tay(x) - - ie., P(Y =) = = x). This paper focuses on the common class-dependent and iirrunce-indeeedennt label noise, i.e., T(x) = T. Usually, the transition matrix T is unidentifiable and hard to learn without additional assumptions [37 5 33
In the literature, many methods try estimate the transition matrix under the anchor point assumption, which assumes that there exist some instances belonging to a specific class almost surely [33 16 (2).
However, the anchor point assumption can not be always satisfied 24 39]. Therefore, methods aiming to develop statistically consistent classifiers without anchor points have been proposed [33 [3]
Among them, the state-of-the-art method (VolMinNet [3] proposed by far the mildest sufficiently scattered assumption, where the anchor point assumption is its special case [T]. They proved that the optimal T is the one with minimum volume enclosing the noisy class posterior of training examples. Hence, they proposed to incorporate the minimum volume constraint of I into label-noiss -nisse learning. To compute the volume of T, the existing method heavily relies on the estimated noisy class posterior. However, the estimation error of the noisy class posterior could usually be large as the deep learning methods tend to easily overfit the noisy labels, especially under the settings of limited training samples 37. This could lead the transition matrix to be poorly estimated when minimizing the volume calculated by using the learned noisy class posteriors. Therefore, how to reduce the side-effects of the inaccurate noisy class posterior has become major concern for making use of the sufficiently scattered assumption.
To address this issue, we propose a forward-backward cocee-consistency regularized algorithm to estimate the transition matrix T. of which we could greatly reduce the dependency of estimating T on using the noisy class posterior to calculate its volume, and further build statistically consistent classifier. Specifically, we show that minimizing the volume of the transition matrix T is equal to maximizing the volume of corresponding clean class posterior. More importantly, maximizing the volume of the clean class posterior could avoid the side-effects of the inaccurate noisy class posterior probability on estimating T.
To be more specific, we will further exploit the equation P(Y x)
TP(Y x). where P = x) E IC T E RCxC, P(Y = x) E RC and C is the number of classes. Note that P(Y X = x) can be estimated by exploiting the noisy data. For simplicity, we re- write the above equation as A = TB for the following derivation, where A and B represent P(Y = x) and P(Y = x), respectively. We adopt the commonly used determinant operation to measure the volume of T, i.e., det(T). As only the square matrix has determinant, we have >=det(TT )det( (B As P(Y = x), i.e., A, can be estimated by the given noisy data, the determinant det(AA is constant. Then, minimizing det(T) (or de((TT) )) equals maximizing B. ). Therefore, we can replace the minimum volume constraint of T by the maximum volume constraint of the clean class posterior P(Y(X=xx)
) BT T
Maximizing the volume of the clean class posterior P(Y = x) can be achieved without directly relying on the estimated noisy class posterior probability. Specifically, project the C-dimensional simplex, which has the largest volume, from the noisy class posterior space onto the clean class posterior space by a diagonally dominant column stochastic matrix T' [13. Let denote the projected clean class posteriors as the backward clean class posteriors P'(y = x). They will make up the largest volume in the sense that they correspond the C-dimensional simplex in the noisy class posterior space. Afterwards, we guide the clean class posterior P(Y = x) learned by the neural network to match to P' P'( = x). Therefore, the volume of the learned clean class posteriors could be maximized to encourage the transition matrix T to converge to its optimal solution. Besides, we further build the cycle-consisteccy regularization between the forward and backward transition matrices T and T' to better learn the clean class posterior probability.
Our main contributions summarized as follows: 1) We propose estimate the transition matrix T under a -bakkadd cycle-consistency regularization, of which we could greatly reduce the dependency of minimizing the volume of the transition matrix T' on the estimated noisy class posterior; 2) We show that such cycle-consistency regularization could help to minimize the volume of the transition matrix T without directly exploiting the estimated noisy class posterior, which encourages the estimated transition matrix T to converge to the optimal solution; 3) Experimental results on four datasets (two synthetic and two real-world datasets) with different label-noise settings consistently justify the effectiveness of the proposed method, on reducing the estimation error of the transition matrix and greatly improving the classification performance.
2

---

Input Image
Backbone Networks
Cross Entropy
T
RY|x
Soltnes
PX
Cross Entropy
T
PYY [x)
Cross Entropy
PYY [X
Traditioal Forward Transtion Motax Estiaation
Barionard Transiion Mattis Estrmation
Porwart: 8 anlowari Cycle Censistency Bequlstizalion
Figure 1: Overview of our method. We optimize the proposed method in an end-to-end manner with three objectives simultaneously. The blue line represents the traditional forward transition matrix T estimation, the green line means the backward transition matrix 7" estimation. The red line means the consistens regularization between the forward and backward transition matrices.
2 Estimating Transition Matrix for Label-Noise Learning
Problem Setting. Let D be the distribution of a pair of random variables (X, Y) (X' x )). where X and Y denote the variable of instances and the corresponding labels, X' E R represents the instance feature space, d is the feature dimension, is the total number of instances, y {1, .C} denotes the label space, and C' is the size of the classes. However, in some real- real-world scenarios, is often expensive or even infeasible to accurately draw large-scale training data independently from the clean distribution D.
Let D be the distribution of a pair of random variables (X, Y) E (Xx)), where Y denotes the variable of noisy labels. In learning with noisy labels, we denote the training dataset 2 {(x, T.) ) which is independently drawn from the noisy distribution D. The goal is learn robust classifier by exploiting the noisy dataset D, which approximates well to the optimal classifier defined on the clean data.
Transition Matrix plays an important role in building statistically consistent classifier through the "loss correction" strategy. It explicitly models the label-noise generation process from clean data distribution to the noisy data distribution, i.e., (xy(x) P(Y = j|Y i, X x), and T(x) CCx It represents the probability that an instance x having the clean label Y = is mislabeled as the noisy label Y j. Intuitively, the clean class posterior probability P(Y|x) can be inferred by the transition matrix Tiy (x) and the noisy class posterior probability P(Y i.e., P(Y)x) = ( x). Note that, in the above equation, the noisy class posterior P(Y x) can be estimated by the noisy data. The transition matrix Txx) is generally unidentifiable and very hard to estimate without additional assumptions. In this paper, we focus on the widely studied class-dependent and instance-inded indepeeednn label noise, i.e, T(x) T, RCXC
Current state-of of- of-thee-rt method (VolMinNet 13 proposes to estimate the class dependet transition matrix T under the sufficiently scattered assumption, where the traditional anchor point assumption its special case. They proved that the optimal T is the one with minimum volume enclosing the noisy class posterior of the training examples. However, in this work, we equivalence problem, i.e., maximizing the volume of corresponding clean class posterior probability, which could reduce the side-effect of the estimation error of the noisy class posterior estimating T. Therefore, to maximize the volume of PYY]X). we propose the forward- bardd--bawdddrr cycle-consistency regularization, and further consistently estimate T'. Specifically, method simultaneously optimize three objectives: the cross entropy loss between the given noisy label i and the estimated noisy class posterior; the cross entropy loss between the backward estimated clean class posteriors and the predicted clean class posteriors by the neural network: the cycle- consistency regularization to minimize the approximation error between P(Y and PYY The backward and cycle-consistency regularization help to minimize the volume of T indirectly without exploiting the noisy class posterior, and further encourage the estimated to converge to its optimal solution.
address its
shown in Fig I our proposed

---

2.1 Forward Transition Matrix Estimation
Given the input instance x, the probability of observing a noisy label Y can inferred as,
C - Σ P 5/Y i)P(Y= = =1/x)
C
P(V= = -1x) =
i=1
PYp = =ix).
We can rearrange Eq. as P(Y = TP(Y(x) under the class dependent and instance indepen- dent assumption, where P(Y = [P(Y = .l,, P(Y = C'x)] and P(Y|x) = [P(Y 1|x),..., P(Y = = _ are the noisy class posterior probability and the clean class posterior proba- bility, respectively. To learn the clean class posterior P(Y[X), we design a neural network f(x; w) parameterized with w, which projects the input instance onto C' classes with a probability output. The transition matrix T should be trainable diagonally dominanl column stochastic matrix, i.e., [ w) to learn the noisy class posterior probability P(Y VolMinNet [I3] shows us that, if f w) models P(Y[X) perfectly and the transition matrix T has the minimum volume, will converge to the optimal transition matrix and the classifier fxx w) will converge to the optimal X.Y In this paper, we introduce the cycle-consistency regularization to indirectly reduce the volume T, without exploiting the estimated noisy class-posterior probability which usually has large estimation error (37]
T [0, 1]cxC Σ T.
and TÜ Tj² for any i + j- Naturally, we could
Given the training dataset D (x,, }.) ji=1+ we optimize the empirical risk by jointly optimizing the transition matrix T and the consistent classifier fx; w) for label-noise learning. Specifically, we minimize the approximation error between the inferred noisy class pesstroostrior probability f w) and the given noisy label ÿ as follows,
min (w, T) w.T
1 N
Σ De log(T T fx; w)),
where N is the number of training examples. Note that, Eq. 2 both the transition matrix and the output of classifier [x,; w) are not given, which makes the transition matrix T unidentifiable without any assumptions. To address this issue, VolMinNet [13 has been proposed by minimizing the volume of the transition matrix. It has been proved that VolMinNet theoretically guarantees the consistencies of both the transition matrix and the classifier.
2.2 Cycle-Consistency Regularization
When minimizing the volume of the transition matrix, VolMinNet exploits the estimated noisy class posteriors to caleulate the volume. Yao et al. (37 has discussed that the estimation error of the noisy class posterior is usually large due to the randomness of label noise. This would lead the transition matrix to be poorly estimated by VolMinNet when the noisy training sample is limited. We will address this issue by proposing a cycle-consistency regularization method instead of minimizing the volume of the transition matrix directly.
As discussed in the Introduction, minimizing the volume of the transition matrix is equivalent to maximizing the volume of the clean class posterior. We show that could directly use the noisy labels to maximize the volume of the clean class posterior rather than using the estimated noisy class posterior calculate the volume of the transition matrix and then minimizing it. Intuitively, we maximize the volume of the learned clean class posteriors by matching it with the projected C-dimensional simplex in the noisy class posterior which also needs to estimate.
More specifically, we first transform the one-hot noisy label into the input noisy class posterior probability by using the SoftMax function, denoting it (oft). Then, we use T" (an approximation of the inverse of the transition matrix) to project onto the clean class posterior space. Since the true clean class posterior is unobservable, here use the output of the designed

---

neural network f(;;; w) as its approximation. Finally, we minimize the following cross- entropy loss between T"SoftMax((.) and f(x;; w). optimize the parameters of w and T simultaneously,
1 N Σ (xx::) SoftMax(j.)).
min min (w,
Note that this objective will help maximize the volume of the learned clean class posterior as T" (o.t has the largest volume in the sense that the noisy labels will make up the C- dimensional simplex. Note also that T' should be initialized by and maintained as diagonally dominant column stochastic matrix, i..., T" E [0, 1|CxC T. and TV > Tj for any i * j.
At present, we can obtain the transition matrices from both the forward and backward ways, i.c.. and T', by minimizing Li and L2, respectively. Intuitively, we should also build an "indirect" cycle-consistency through minimizing the difference between P(Y =x) where "indirect" means that we would make use of the invertible relationship between these two matrices T and T" indirectly through the original and circularly computed clean class-posterior. Specifically, the cycle-consistency regularization can be expressed as,
x) and (T x)),
min Laww; T,T"
Σ f(x:: log(T f(x;:))). N
1
(4)
Note that, in Eq. 4 we just optimize the parameter w in the network classifier (x,; w), while keep T and T' be constant. This is because that our ultimate goal is to learn good clean class posteriors. If the transition matrices are well learned, the estimated clean class posterior should satisfy this constraint.
Based on the above descriptions, the overall objective function can be expressed as,
L L1 (w, T) (w, T") AC3 Tw;")"",,
(5)
where À the trade-off hyper-paramettr balance the two transition matrix learning objectives with the cycle-consistency regularization term. We optimize all the model parameters simultaneously in an end-to-end manner.
2.3 Theoretical Analysis
In this section, we justify that the proposed method helps to minimize the volume of the transition matrix T without directly relying the learned noisy class posterior, which does not prevent the estimated from converging to its optimal solution.
As theoretically proved in VolMinNet [33, the optimal transition matrix T is the one with minimum volume enclosing the noisy class posterior of the training examples under sufficiently scattered assumption. VolMinNet thereby theoretically guarantees the consistencies of both the transition matrix and classifier by minimizing the volume of the transition matrix. Under the same sufficiently scattered assumption, our proposed method also theoretically guarantees the consistencies of both transition matrix and the classifier because, in Introduction, we show that minimizing of the transition matrix is equal to maximizing the volume of the clean class posterior. Importantly, our method could help reduce the sidc-cffects the estimation error of the noisy class- dlas-posterior probability.
volume
Specifically, our proposed framework as shown in Eq. 5 simultaneously optimize the forward and backward transition matrices T and T'. Especially during T" estimation, the input is the given one-hot noisy label which corresponds to the simplex having the maximum volume. When they pass through the diagonally dominant matrix T", the projected clean class-posteria probability should intuitively has a large volume. Then, minimizing the cross entropy between the learned clean class posteriors and the projected clean class posteriors could help maximize the volume of the learned clean class posteriors. Note that the cycle-consistency constraint on T and T" as illustrated in Eq. further helps to learn better clean class posterior the projected clean class posterior is more accurate.
3 Experiments
In this section, we introduce the experiment setup, including datasets, noise types, and implementation details. We compare our proposed method with the state-of-the-art algorithms on two synthetic and
5

---

Table I: Comparison with sate-of-the--a methods CIFAR- 10 and CIFAR-100 datasets. The mean and standard deviation computed over five runs are presented. "Sym-xx%" means the noise rate is xx% and noise type is "Symmetry".
Cifar-10
CIFAR-
Method
Sym-20%
Sym-10%
Sym-60%
Sym-20%
Sym-40%
Sym-60%
(baseline)
84.58 0.18
80.78
68.31 0.33
51.93 =0.39
40.11= 0.86
25.81 0.74
GCE [4]
89.31 +0.07
86.61 0.23
79.40 0.41
66.07 0.24
59.03= 0.21
45.68 0.39
PeerLoss 17
88.78 +:118
84.87+ 0.15
75.28+ 0.31
57.34= 0.34
43.39= 0.33
28.66- 0.67
Co-teaching 6
85.76 +0.26
83.12+ 0.31
70.89+ 1.06
56.83+ 0.28
43.38- 0.51
28.04+ 0.69
Co-teaching++ + 38
86.3910.33
83.80=0.30
72.5110.46
57.64110.34
44.28:10.83
29.60=1.116
T-Revision
88.01=0.16
84.52=0.11
71.53:10.22
62.66:10.53
55.25:10.36
39.9411.28
VolMinNet 13
89.69=0.19
85.46:20.19
73.55++0.78
64.70:1:0.0
56.25-+0.45
41.06=0.45
DualT [37
89.8810.13
86,2310.64
72.2111667
65.75.10.38
56.8011.118
42.56
T-For (7')
89.53+0.11
85.38=0.13
73.01+0.54
64.234+0.64
56.02+00.99
40.89+0.37
T-Back (?")
88.40:10.12
84.9710.16
73.1210.79
63.39:10.2
54.96:10.43
41.15=0.82
T T'
89.6410.16
85.4710.32
73.39110:0
64.95110.1
56.36100.51
41.9410.43
Ours
90,44=1.119
87.30
81.01
67.74.00.17
61.71+10.20
49.30-10.82
two real-world noisy datasets, followed by an ablation study to analyze the experimental results and some useful hyper- spper-parameters.
3.1 Experiment Setup
Datasets. Extensive experiments are conducted on two manually corrupted datasets with different noisy types (i.e., CIFAR-I0 [9), CIFAR- 100 [9]) and two real-world noisy datasets ( (i... Cloth- ing IM 34 and Food-1011 101N (0),, demonstrate the effectiveness of the proposed method. Both CIFAR-10 and 110 contain 60K images of size 32 x 32, of which 50K images constitute the training set and 10K images for testing set, while CIFAR-10 contains 10 classes, and CIFAR-100 contains 100 classes, Clothing M is real-world noisy dataset that contains 1M images with about 38.46% noisy labels for training and 10K images with clean labels for testing. Food- 101N is also real-world noisy dataset which contains 310K images with about 19.66% noisy labels for training and 55K images with clean labels for testing.
Noisy type. For CIFAR-I0 10 and CIFAR-100, we manually corrupted the training set according to the ground-truth transition matrices. Specifically, we conducted experiments using three commonly used noisy types: (I) Symmetry flipping [20 ; (2) Asymmetry flipping [2]: (3) Pair flipping (8);
3.2 Implementation Details
For fair comparisons, all our experiments are performed NVIDIA GeForce RTX 3090, and implemented on the same PyTorch platform. For CIFAR-10 and CIFAR- -00, the backbone we used is ResNet-34. We train the classification network [x:; w), the transition matrices T and 7" by SGD strategy, with batchsize of 128, momentum 0.9, weight decay 10-3, and learning rate 10-2 For CIFAR-10, the algorithm run 60 epochs and the learning rate is divided by 10 after the 30-th epoch. For CIFAR- 100, the algorithm run 80 epochs and the learning rate is divided by 10 after 30-th and 60-th epoch. For Clothing 1M and Food-101N, the backbone is pre-trained on ImageNet. We train the classification network / fx;;; w). the transition matrices T and T' also with SGD strategy, with batchsize of 32, momentum 0.9, weight decay 10-3, and learning rate 2 x 10-3 The algorithm run 80 epochs and the learning rate is divided by 10 every 30 epochs. Before training, we warm up on all noisy data with early stopping technique, where we have trained 10, 10, and epochs on the CIFAR-10, CIFAR-100, Clothing 1M and Food 10IN datasets, respectively.
used is ResNet-50 which
3.3 Comparison with state-of-the-art methods
We compare the proposed method with the following representative works: 1) CE, which trains the classification network with the standard cross-entropy loss on the original noisy dataset; 2) GCE 40
3) PosrLosss [7]: 4) Co-teaching 6 5) Co-teaching++ [38]; 6) T-Revision [3); 7) Dual-T
37
VolMinNet [3] which estimates the transition matrix under sufficiently scattered assumption.

---

Table 2: Comparison with state-of-the aht methods CIFAR-I0 and CIFAR-100 datasets. The mean and standard deviation computed over five runs are presented. "Asym-xx% means the noise rate is xx% and noise type is "Asymmetry".
Cifar-10
CIFAR-100
Method
Asym-20%
Asym-10%
Asym-60%
Asym-20%
Asym-40%
Asym-60%
CE (baseline)
84.7111.22
81.2610000
68.40-1.16
52.16.0.37
40.2110.23
26.56:10.64
GCE [40
89.5410.21
85.9510.40
79.5
63.66-10.3
57.3410.35
45.46:10.16
PeerLoss
77
88.9810.15
85.61
77.03=0.49
57.5110.05
43.95.0.355
30.02:10.39
Co-teaching 6
05.90+0.38
83.09+0.44
71.69=0.50
0..27
43.76+0046
30.18+0.71
Co-teaching++
38
87.1310.07
84.6610.36
73.50=0.47
58.79+0.35
45.2610.41
32.02::1.22
T-Revision 33
87.99+0.32
85.17=0.07
72.93=0.27
63.94+00119
1..19+1.28
42.36++1099
VolMinNet 13
89.62+0.15
16.12+0.16
74.80-0.15
65.91+0.25
58.35+0.35
42.16+0.94
DualT [37]
89.36110.44
86.59-0.30
78,89=0.99
65.76-10.56
56,90.0.399
4..61
T-For (7)
89,46+0.21
85.74+0.11
74.54=0.12
65.30++0.01
56.31+0.42
4..21
T-Back (T")
89.9710.14
85.81
73.40-0.81
64.56.10.34
$5.09
41.73:10.73
T + T
89.62.10.24
86.25-0.03
74.80-0.21
65.52:10.28
$7.1010.200
42.72:10.29
Ours
90.55++0.03
87.29+00.05
82.58-0.24
68.34
62.64+0.49
50.29
Table 3: Comparison with tfae-oofthheart methods on CIFAR-10 and CIFAR-100 datasets. The mean and standard deviation computed over five runs are presented. "Pair-xx% means the noise rate is xx% and noise type is "Pair".
Cifar-10
CIFAR-100
Method
Pair-20%
Pair-10%
Pair-15%
Pair-20%
Pair-10%
Pair-15%
CE (baseline)
86.26:+0.41
80.2440.61
70.73=1.44
52.85+0.35
39.66=40.68
28.06:10.28
GCE [40
89.7710.17
84.11:0.11
77.54=0.81
65.37=10.29
50.9110.64
42.38:10.12
PeerLoss 77
90.1110.03
85.52-0.440
76.75
60.48110088
44.5510.29
39.51
Co-teaching 6
86.81+0.31
82.29+0.23
74.34-0.74
59.54+0.14
44.97+0.49
39,59+0058
Co-teaching++ 38
07.44+0.33
83.59+0..3
73.96=0.35
03.58
46.76+00.3
42.60:t0.72
T-Revision 33
90.90:10.11
87.03-10.37
1.18
66.37:10.5
44.64
38.84:10.47
VolMinNet 13
90.99+10.27
88.97=0022
74.77=2.15
69.70=+0.30
39.23++0.17 4..63
DualT
90.1810.39
88.81
77.22=22008
70.07::0.2
53.15
02.58:10.54
T-For (?)
90.25+0.40
88.40+0.35
74.08+ 1.88
69.2740.14
40.65
39. 19.00
T-Back (T')
90.0311.11
87.0910.93
73.2610.89
68.6110.119
44.4110.31
38. 48.86400.44
T T'
90.67+0.27
89.35+0049
78.62+1.40
6..50
44.79+0.65
39.16 19.16+0.55
Ours
91.67+00.27
91.36+0.13
91.08-0.00
71.63+0.39
70.87+0.14
69.18+1.30
Results on the synthetic noisy datasets. Tables 1 and report the experiment results under three different synthetic noisy types on cifarl( and cifar datasets, respectively. Each table compares the proposed method with representative works on one synthetic noisy type. As illustrated in Eq, 5 our overall objective contains three variants as described in the tables: 1) "T-For (T)" means that we optimize the classifier just use Li(w, T);
: 2) "T-Back (T')" means that we optimize the classifier
just use (2(w, T'): 3) "T + T'" means that we optimize the classifier use Li (w, 7) + L2 T 4) utilize the overall objective shown in Fq. 5o optimize the classifier. Compared with the representative methods, the proposed method achieves the best performance on both datasets with three synthetic noise types under three different noise ratios. The evaluation results of the three tables can be summarized as follows:
"Ours" means that
Compared with representative methods, the proposed method has significant improvements over these methods, which specifically outperforms the baseline method "CE" and the representative work VolMinNet [13 by large margin of 16.80%and 7.24% average.
As the noise rate increases, the superiority of this method gradually emerges. As shown in these three tables: for symmetric noise, the average advantage of our method over other methods is 4.47% 6.65% and 10.54% at noise rate of 20%, 40% and 60%, respectively; For asymmetric noise, the average advantage of our method is 4.46%, 6.61 % and 10.28% at noise rate of 20%, 40% and 60%, respectively; For pair noise, the average advantage of our method 4.61% %, 15.31 and 22. 88% at noise rate of 20%, 40% and 45% respectively.
This indicates that method can handle difficult cases with high noise rates much better.
7

---

Table 4: Classification accuracy (%) on the Clothing 1M dataset. (*) indicates that the implementation of the compared method is based on the authors' code.
Methods
CE (Baseline)
GCE [40
SL [26]
Co-teaching 6
JointOpt [3
Lparr [35] 72.46
Accuracy
68.94
69.75
71.02
69.21
72.16
Methods
PTD R.D.V.
32
ERL 15
ForwardT 20
JoCor [27 70.30
CORES S 73.24
CAL [1 74.17
Accurncy
71.67
72.87
69.84 84
Methods
MEIDTM A
VeeMinNet*
Outs 70.73
DivideMix* 74.67
DivideMix+ VolMinNet
DivideMix+OOus 75.12
Accuracy
73.05
69.82
74.83
Table 5: Classification accuracy (%) on the Food-101N dataset. (*) indicates that the implementation of the compared method is based on the authors' code.
Methods
CE (Baseline)
Nleyn
10
CleanNetare
10
DeepSelf y
NoiseResist
VolMinNer* [X] 83.04
Accuracy
81,44
83.47
83.95
85.11
84.70
Methods
DivideMix
Ours 83.71
DivideMix- -T-Foe-TTTFo DivideMixx- (T")
DivideMix VolMinNer
DivideMixxOurr: 86.11
Accuracy
84.39
85.07
84.83
85.07
By comparing the four variants of our method, it clearly shows that integrating the backward transition matrix and cycle-consistency regularization can greatly help to improve the baseline performances step by step.
Results on the real- wolld datasets.. Tables 4 and 5 compare the proposed method with representative works on Clothing M and Food-101N I01N datasets, respectively. We can clearly see that our methods outperforms the baseline method "CE" and VolMinNet [I3] by margin of 1.79% and 0.91% on Clothing IM dataset, and 2.27% and 0.67% on Food- 101N dataset. What's more, since the proposed method can be used as plug-and-play module, we integrate this module into the representative work DivideMix [T] and denoted as "DivideMix+Ours" To further illustrate its effectiveness, we also combine DivideMix with VolMinNet as another baseline, denoted as "DivideMix ViviieMii+VVlMMiNNe"" VolMinNet". We can clearly see that our method achieves superior performances over all these methods.
3.4 Ablation Study
As shown in Figure 2 (a) to we also show the estimation error of the transition matrix T of our method during the model training, with different noise types and different noise rates, the CIFAR- 10 and CIFAR-100 datasets, to evaluate the estimated transition matrix T. The estimation error is measured by the 4 norm between the ground-truth transition matrix and the estimated transition matrix. It shows that the proposed method always outperforms previous algorithms on T estimation. To explore the effect of Syper--arameter on model performance, we conducted experiments with different À under different noise types and noise rates on CIFAR- 10 and cifar- 100 datasets, as shown in Figure 2 (m) to (p) Each experiment run five times. We summarize Figure 2 (m) [p]aas as follows: 0.3, the estimation error the smallest and we get the best performances. Based on this observation, we set as 0.3 in all our experiments.
1) The smaller the T estimation error is, the higher the test accuracy will be: 2) When
4 Related Work
Based on the statistical consistency of the learned classifier, we roughly divide existing label-noise learning methods into two categories: algorithms with statistically inconsistent classifiers (i.e., heuristic algorithms), and the algorithms with statistically consistent classifiers.
The statistically inconsistent algorithms usually explore some heuristics to reduce the side-effecct of the noisy labels. The representative works include: 1) the data cleaning methods which specially design some strategies to select reliable examples [18 5 2) the "label correction" methods which
aim to improve the label quality during model training 23 21:; 3) the supervipee learning methods which treat the unreliable examples as unlabeled data and then adopt some self- self-supervised training technics for robust feature representation learning [II; 4) some other classic machine learning techniques [19 20), such as soft label based methods (29] early stop tricks to avoid over- fitting 2 RT Although these methods empirically work well without explicitly modeling the label noise distribution, can not theoretically guarantee the consistency of the learned classifiers (6).
8

---

C1FAB S.
CTEAR 16
CIFAR
Oan
lanee
(
(b)
(c)
(d)
(YEAB-IX Anta43
0.30
Forton
1peck
Epich
Epech
(e)
(f)
(g)
(h)
CIPAN-IE, PaR-07
CHAMMAN Parn
CIAAR-1BA Phardd
Soree
kpeck
Fpmch
Epach
(k)
CHAAB-I P
CHPSE Parts
CIAAB-35, Psb-4
CUPAR 191 PiareLL
Eyech
(m)
(n)
(o)
(p)
(a) to () compare the estimation error of T between our method and other transition matrix based methods with different noise types and noise ratios two datasets. [m) to show
Figure 2:
classification accuracy and T estimation error with various values of X on two datasets.
The statistically consistent algorithms aim learn consistent classifier with the noisy data, which will asymptotically converge to the optimal classifier obtained on its corresponding clean r37. These methods are primarily designed based on the "loss correction" strategy. Among them, the transition matrix T(x) has been used to modify the loss function to build consistent classifiers. Specifically, the anchor point assumption has been widely adopted to estimate the transition matrix [16 33), and some methods directly assume that the anchor points have already been given. However, this assumption sometimes could be too strong in real application. When no anchor point is given in the dataset, these al algorithms can not be guaranteed statistically consistent. Then, to remove this strong dependence on anchor points, some methods propose the slack variable trick (25), which just use the data points with high noisy class-posterior probabilities to estimate the transition matrix.
Afterwards, series of assumptions 4 13 have been proposed to efficiently estimate the transition matrix. For example, Dual-T estimator transition matrix into the product of two casy-to-estimate transition matrices. ExtendedT [30] proposed to extend the traditional transition matrix to be able to model mixed close-set and open-set label noise. VolMinNet [13] try to consistently estimate the transition matrix under the sufficiently scattered assumption, which empirically incorporates the minimum volume constraint of T into the label-noise learning. However, these methods relies heavily on the estimated inaccurate noisy class-posterio probability, which could lead the transition matrix to be poorly estimated.
[37] introduces the intermediate class to factorize the original
5 Conclusion and Limitation
Estimating the transition matrix T plays an important role building statistically consistent classifier for label-noise learning. To address the bottleneck of how to reduce the sidc-effects of the large

---

estimation error of the noisy class posterior on transition matrix estimation, this paper proposes to cyclecconsittncyy regularization, of which we could greatly
estimate T under a reduce the dependency of estimating T on the estimated noisy class posterior. We also show that the proposed method helps to minimize the volume of T without directly exploiting the estimated noisy class posterior, which encourages the estimated transition matrix T to converge to its optimal solution. Extensive experimental results consistently justify the effectiveness of the proposed method, from both of the superior classification accuracy and the estimation error of the learned transition matrix. Limitation. One major limitation in this study is that our method is constrained to work well under the sufficiently scattered assumption. When this assumption is violated in some cases, the statistical consistency of the estimated transition matrix and classifier would not be guaranteed. In the future, will make deep theoretical analysis on the backward transition matrix and estimated clean class posterior probability.
This work was supported in part by the National Natural Science Foundation of China under Grant 62176198, 61922066 and 62 106184 part by the National Key Research and Development Program of China under Grant 201 8AAA0103202. in part by the Technology Innovation Leading Program of Shaanxi under Grant 2022QFY01-15, in part by Open Research Projects of Zhejiang Lab under Grant GOABOO.. Tongliang Liu was partially supported by Australian Research Council Projects DP180103424, 19-19011,473, IC- 190100011.1, DP-222222222211111 121, and FT-220100318. Bo Han was supported by the RGC Early Career Scheme No. 22200720, NSFC Young Scientists Fund No. 62006202, and Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011655
Acknowledgements:
References
[1] Devansh Arpit, Stanistaw Jastrzębski. Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, al. A closer look memorization in deep networks. In International conference on machine learning, pages 233 242. PMLR, 2017.
[2] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang. Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Un- derstanding and improving early stopping for learning with noisy labels. arXiv preprint 1r6: 15853, 2021.
[3] Avrim Blum, Adam Kalai, and Hal Wasserman. Noise-tolerant learning, the parity problem, and the statistical query model. Journal of the ACM (JACM), 50(4):506-51) 519, 2003.
[4] De Cheng, Tongliang Liu, Yixiong Ning. Nannan Wang. Bo Han, Gang Niu, Xinbo Gao, and Sugiyama. Masashi. eestaacce-deeedeee label-noise learning with reguuforldreeulrrizeedd transition matrix estimation. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, 2022.
[5] Hao Cheng. Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. ICLR, 2021.
[6] Bo Han. Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-tcaching: Robust training of deep neural networks with extremely noisy labels. arXiv preprint arXiv:1804.0660 2018
[7] Jiangfan Han, Ping Luo, and Xiaogang Wang, Deep self-learning from noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5138-5147, 2019.
[8] Lu Jiang, Zhengyuan Zhou, Thomas I Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pages 2304-2313 2313. PMLR, 2018
[9] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[10] Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5447-5456, 2018.
[11] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint 270774200,..73344 2020.
[12] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019.
10

---

[13] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. arXiv preprint a1XIv:2102.02400, 2021.
[14] Chang Liu, Han Yu, Boyang Li, Zhiqi Shen, Zhanning Gao, Peiran Ren, Xuansong Xie, Lizhen Cui, and Chunyan Miao. resis-resistant deep metric learning with ranking- based instance selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6811-6824 2021.
[15] Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fermandez-Grandaa Early- regular- ization prevents memorization of noisy labels. arXiv preprint arXiv:2007.00/5// 2020.
[16] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):4474411 2015.
[17] Yang Liu and Hongyi Guo. Peer loss functions: In International Conference on Machine Learning, pages 6226- 6236. PMLR, 2020.
Learning from noisy labels without knowing noise rates.
[18] Yueming Lyu and Ivor W Tsang. Curriculum loss: Robust learning and generalization against label corruption. arXiv preprint arXiv:1905. 10045, 2019.
[19] Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijewickrema, and James Bailcy. Dimensionallty-driien learning with noisy labels. In International Conference on Machine Learning, pages 3355-3364, PMLR, 2018.
[20] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944-1952, 2017.
[21] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.66 2014.
[22] Clayton Scott. rate convergence for mixture proportion estimation, with application to learning from noisy labels. In Artificial Intelligence and Statistics, pages 838-846. PMLR, 2015.
[23] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3552-5560, 2018
[24] Robert A Vandermeulen and Clayton D Scott. An operator theoretic approach to nonparametric mixture models. The Annals of Statistics, 47(5))2704 223323347773,, 2019.
[25] Xiaobo Wang, Shuo Wang, Jun Wang, Hailin Shi, and Tao Mei. Co-mining: Deep face recognition with noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9358-9366 3358-9367,22 2019.
[26] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 322-330, 2019.
[27] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agrecment: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13726-13735, 2020.
[28] Songhua Wu, Xiaoho Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: new perspective on learning with label noise. 2020.
[29] Yichen Wa, Jun Shu, Qi Xie, Qian Zhao, and Deyu Meng. Learning to purify noisy labels via meta soft label corrector. AAAI, 2021.
[30] Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, Yinian Mao, and Tongliang Liu. Extended T:learning with mixed closed-set and open-set noisy labels. IEEE Trunsactions on Pattern Analyxis and Machine Intelligence, 2022.
[31] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong. Nannan Wang, Zongyuan Ge, and Chang, Robust early-learning: Hindering the memorization of noisy labels. In International conference on learning representations, 2021.
[32] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part- label noise: Towards instance dependenn label noise, Advances in Neural Information Processing Systems, 33, 2020.

---

[33] Xiaobo Xia. Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32:6838-688 68-9899, 2019.
[34] Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2691-2699, 2015.
[35] Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L_dmi: A novel information-theorrii loss function for training deep nets robust to label noise. In NewrlPS, pages 6222-6233, 2019.
[36] Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine learning. 95(3):291-327, 2014.
[37] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong. Jiankang Deng. Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error transition matrix label-noise learning. arXiv preprint arXiv:2006.0780 07800 2020.
[38] Xingrui Yu, Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pages 7164 7164-7173. PMLR, 2019.
[39] Xiyu Yu, Tongliang Liu, Mingming Gong, Kayhan Batmanghelich, and Dacheng Tao. An efficient and provable approach for mixture proportion estimation using linear independence assumption. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4480 4489, 2018.
[40] Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
[41] Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- deaendeed lahel noise. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 10113-10123 2021. 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including thcorctical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasnng new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're us-
ing/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
12

---

5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include full text of instructions given to participants and screenshols, if applicable?
[N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
A
Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in the appendix. This section will often be part of the supplemental material.
13


Reviews:
[{"review_id": "yrYAgR20Wj5", "reviewer": "Reviewer_f3fT", "summary": "This paper proposes a label-noise learning algorithm through estimating the transition matrix T, to build a statistically consistent classifier. Specifically, the proposed algorithm tries to estimate the transition matrix under the forward-backward cycle-consistency regularization, which helps to minimize the volume of the transition matrix indirectly without exploiting the estimated noisy class posterior. Theoretical analysis and experimental results justify the effectiveness of the proposed method.", "questions": "see the weaknesses above", "limitations": "YES", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1)\tThe proposed method is technical sound and novel. It develops an algorithm to estimate the transition matrix under the sufficiently scattered assumption, by incorporating the proposed cycle-consistency regularization. This method can reduce the dependency of estimating the transition matrix T on the inaccurately estimated noisy class posterior.\n2)\tDetailed theoretical analysis illustrates that the cycle-consistency regularization helps to minimize the volume of the transition matrix T, which encourages the estimated T to converge to its optimal solution.\n3)\tThe experiment is sufficient. They illustrate the effectiveness of the proposed method from both of the classification performance and the T estimation error, on both the synthetic and real-world noisy datasets.\nWeaknesses:\n1)\tSince this method needs a forward-backward training process, does it require more time to train the network?  How many extra parameters are introduced in the newly proposed method compared with the previously proposed method VolMinNet[13]?\n2)\tThis paper states that it tries to estimate the transition matrix under the sufficiently scattered assumption, what’s the difference between this assumption and the previous anchor point assumption?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mBOiYivWveG", "reviewer": "Reviewer_3zVp", "summary": "The paper proposes a method to achieve good prediction accuracy under label noise, by estimating the transition matrix from clean conditional probabilities, to noisy conditional probabilities. The paper presents experiments on synthetic noise models with a known transition matrix and natural noise models.\n", "questions": "- What is the motivation for the method proposed in section 2.2? For instance, why is it necessary to learn the second matrix $T^b$? \n\n- What happens if one only minimizes the second term in equation 5? Some of the experiments imply good performance and it would be interesting to understand what is the reason behind the method’s success in those scenarios. A more thorough ablation study would likely reveal what makes the method work well and what is perhaps unnecessary and can be stripped away from the loss.\nIt is difficult to assess the performance of the method because crucial details about the experimental setup are missing. What are the exact noise models? How many classes are assumed to suffer from label noise in the symmetric/asymetric/pair model? What was the procedure used for hyperparameter tuning for the baselines and for the proposed method?\n\n- The method seems to not perform particularly well on datasets with natural noise. Why are only some and not all baselines presented in tables 4 and 5? What are the confidence intervals for the values in tables 4 and 5? What are possible reasons why the proposed method performs worse on natural noise models compared to the synthetic noise instances? I suggest adding to the experimental comparison more settings with noise models other than the ones considered in Tables 1-3, e.g. random flips for all classes with equal probability, other natural label noise datasets (e.g. OpenImages, MS-COCO with the noisy annotations, the dataset of https://github.com/zhongyy/Unequal-Training-for-Deep-Face-Recognition-with-Long-Tailed-Noisy-Data etc).\n\n- How does a simple baseline that uses early stopping regularization (see e.g. https://arxiv.org/pdf/1910.09055.pdf and http://proceedings.mlr.press/v108/li20j/li20j.pdf) compare to the proposed method?\n", "limitations": "It is unclear from the results reported in the tables (in particular Tables 2 and 3) when the proposed method breaks and what exactly are the assumptions that it requires to perform well. Instead of selecting the 3 fixed noise rates reported in the paper, I suggest a plot in which the noise rate is varied on the Ox axis, while the Oy axis indicates the test accuracy. See also the discussion regarding other noise models in the Questions section.\n", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Some of the experimental results are promising, especially on the synthetic noise models (Tables 1-3 and Figure 2).\n\nWeaknesses:\n\n- The paper could benefit from some changes regarding its style, which is atypical and unfortunately hurts clarity quite significantly. Here are just a few examples of things that could be improved. The introduction contains heavy notation that is not necessary at this point. The introduction even contains the proof of the “theoretical result”, as the paper notes on line 188. The paper is also not self-contained, and important concepts are not defined at all, e.g. the sufficiently scattered assumption or the anchor point assumption, the sym, asym or pair noise models, the baselines etc. In addition, the writing is at times too colloquial (e.g. “we creatively propose”, “what’s more” etc).  There are also numerous typos and grammar mistakes (line 39, 53 etc).\n\n- The paper does not explain how the authors arrived at the procedure described in section 2.2. (see also the Questions section). There is also very little discussion on the impact of the 3 terms in the loss, or why the loss in equation 5 is particularly well suited for this problem.\n\n- It is not possible to review the soundness of the experimental evaluation since too many details are currently missing from the manuscript. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c4jY7sh1pan", "reviewer": "Reviewer_zugN", "summary": "This paper proposes the idea of using the forward-backward cycle-consistency regularization to estimate the transition matrix T. This method addresses a long-standing problem of estimating the transition matrix depending on the inaccurately estimated noisy class posterior probability. Theoretical analysis and experimental results both show that the proposed method is superior to the compared methods. Particularly, it seems that this paper firstly proposes to estimate the transition matrix T in a backward manner.", "questions": "Besides answering the above listed weaknesses, I also curious about the following question: Why not directly use T^-1 instead of the learned T^b to optimize the transition matrix?", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) Novelty: The main contribution of this work is the idea of estimating the transition matrix T with the forward-backward cycle-consistency regularization, under the sufficiently scattered assumption. Maybe, this is the first to estimate the transition matrix T bidirectionally.\n2) Writing: The paper looks good and well-written. The introduction and the algorithm part are easy to follow.\n3) Experiments: The proposed method achieves better results in both the synthetic and real-world datasets, and provides detailed ablation study.\n\nWeaknesses:\n1) It needs some model complexity analysis, especially for the training process of the backward and forward transition matrix.\n2) Some implementation details are missing. How do you combine the proposed transition matrix estimating approach with the traditional DivideMix[11] algorithm, as shown in Table 4 and Table 5 ?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PuN27q2z9E", "reviewer": "Reviewer_4D9u", "summary": "This paper focuses on the class-dependent noisy labels. To solve this, the authors proposes a cycle-consistency regularization on the estimation of the transition matrix for learning with class-dependent noisy-labels. The proposed method could encourage the estimated transition matrix to converge to its optimal solution, without explicitly estimating the noisy class posterior probability. Therefore, it could help to build a better statistically consistent classifier. Experimental results on several datasets show the effectiveness of this method, on reducing the estimation error of T and boosting the classification performance.", "questions": "See the weaknesses above.", "limitations": "The authors have discussed the limitations of this paper, and there is no negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Different from the previous representative work VolMinNet, this paper proposes a new strategy to optimize the transition matrix T. The proposed method successfully addresses the problem of how to reduce the side-effects of the inaccurate noisy class posterior on estimating the transition matrix T. \n2. The author conduct experiment on several datasets to show the effectiveness of the proposed method. Moreover, the theoretical analysis is also provided to demonstrate the effectiveness of the algorithm.\n\nWeaknesses:\n1. In the overall objective function shown in Eq. (4), I believe we could also obtain the experiment results by just optimizing the backward transition matrix T^b. That is to say, we could just minimize Eq. (3) to obtain one intermediate results. However, this paper does not list this result for experimental ablation study.\n2. In line 43-44, why anchor point assumption is a special case of the sufficiently scattered assumption? \n3. The author claimed that the proposed method can be used as a plug-and-play module and integrate it into the DivideMix. The authors should clearly define the experiment settings and how they are combined. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept