Improving the Estimation of Instance-dependent Transition Matrix by using Self-supervised Learning
Authors: Yexiong Lin
Yu Yao
Zhaoqing Wang
Xu Shen
Jun Yu
Bo Han
Tongliang Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: The transition matrix reveals the transition relationship between clean labels and noisy labels. It plays an important role in building statistically consistent classifiers. In real-world applications, the transition matrix is usually unknown and has to be estimated. It is a challenging task to accurately estimate the transition matrix, especially when it depends on the instance. Given that both instances and noisy labels are available, the major difficulty of learning the transition matrix comes from the missing of clean information. A lot of methods have been proposed to infer clean information. The self-supervised learning has demonstrated great success. These methods could even achieve comparable performance with supervised learning on some datasets but without requiring any labels during the training. It implies that these methods can efficiently infer clean labels.
Motivated by this, in this paper, we have proposed a practical method that leverages self-supervised learning to help learn the instance-dependent transition matrix. Empirically, the proposed method has achieved state-of-the-art performance on different datasets.

Under review as a conference paper at ICLR 2023
IMPROVING THE ESTIMATION OF INSTANCE- DEPENDENT TRANSITION MATRIX BY USING SELF- SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The transition matrix reveals the transition relationship between clean labels and noisy labels. It plays an important role in building statistically consistent classi- fiers. In real-world applications, the transition matrix is usually unknown and has to be estimated. is a challenging task to accurately estimate the transition matrix, especially when it depends on the instance. Given that both instances and noisy labels are available, the major difficulty of learning the transition matrix comes from the missing of clean information. infer clean information. The self- learning has demonstrated great suc- cess. These methods could even achieve comparable performance with supervised learning on some datasets but without requiring any labels during the training. It implies that these methods can efficiently infer clean labels. Motivated by this, in this paper, have proposed practical method that leverages self-supervised learning to obtain nearly clean labels to help the learning of instance-des deneede-deeennent transition matrix. Empirically, the proposed method has achieved state-of-the-art performance on different datasets.
lot of methods have been proposed
INTRODUCTION
Recently, more researchers the deep learning community place emphasis on learning with noisy labels (Jiang et al. 2018 Liu 2021 Yao et al. 202 Bai et al. 2021 Ciortan et al. 2021). This is because manually annotating large-scale datasets labor-intensive
con--conuuming. Then some cheap but imperfect methods, e.g., crowdsourcing and web crawling, have been used to collect large- scale datasets which usually contain label errors. Existing work shows that training deep learning models on these datasets can lead to performance degeneration, because deep models can memorize the noisy labels easily (Han et al. 20188) Bai al.] [00T]. How to improve the robustness of deep models when training data containing label errors becomes important research topic.
To learn a classifier robust to label noise, there are two streams methods, i.e., statistically in- consistent methods and statistically consistent methods. The statistically inconsistent methods
mainly focus on designing heuristics to reduce the negative effect of label noise (Nguyen et al.
2019 Li et al. 2019 2020 Wei et al. 2020 Bai et al.] 2021]; Yao al.) 2021a). These meth- ods have demonstrated strong empirical performance but usually require expensive hyper-parameter tuning and do not provide statistical guarantees. To address the limitation, another stream of meth-
ods focusing on designing classifier-consistnn algorithms (Liu Tao 2013) Patrini et al. 2017]
Xia al. 2020: Li et al. 2021) by exploiting the noise transition matrix T(x)
RCxC where
(x) P(Y = ,|Y = i, z). X denotes the random variable for instances or features, Y
denotes the noisy label, Y denotes the clean label, and C' denotes the number of classes. When
transition matrix is given, the optimal classifier defined on the clean domain can be learned by
utilizing noisy data only (Liu & Tao 2015 Xia et al. 2019).
In real-world applications, the destance transition matrix T(x:) usually unknown and has to be learned. It is still challenging task to accurately learn T(z) (Li et al. 2019 Yao et al. 2020). The reason is that to accurately learn T(x), the instance X, the noisy label Y and the clean label Y generally have to given. However, for the dataset containing label errors, clean labels usually not available. In general, without any other assumptions, learn the transition matrix

---

Under review as a conference paper at ICLR 2023
for instance, its clean-label information has to be given. Then existing methods hope some clean- label information can be inferred to learn ((r) (Xia et al. 2019 Yang et al. 2022 Li et al. 2021
We will discuss the details in Section 2
Recently, the classification model based on self-supervised learning has demonstrated comparable performance with supervised learning some benchmark datasets (He al. 2020 Niu et al. 2021]. This implies that self-supervised learning has a strong ability to infer clean labels. Motivated by this, in this paper, we propose CoNL (Contrastive label-Noise Learning), which leverages the self-supervised technique to learn the instance-dependent transition matrix. In CoNL, it contains two main stages: contrastive co-selecting and constraint (x) revision which are as follows:
We propose contrastive co-selecting, which utilizes the visual representations learned by contrastive learning to select confident examples without employing noisy labels. In such a way, the learned visual representations will be less influenced by label errors. To select con- fident examples defined on the clean domain, we learn two classifiers estimating P(Y and two transition matrices simultancously by employing noisy labels and learned repre- sentations. We also encourage the two classifiers to have different learning abilities by training them with the representations obtained from strong and weak data augmentations, respectively. Then they can learn different types of confident examples and be robust to different noise rates. Combining two classifiers can obtain more confident examples.
We propose constraint T(x) revision, which refines the learned transition matrix by em- ploying the selected confident examples. Based on the philosophy that the favorable tran- sition matrix would make the classification risks on both clean data and noisy data small. We fine-tune T(x) by encourage the loss w.r.t. the estimated P(Y ()) and the estimated X) be small on the selected confident examples.
The empirical results for both transition-matrix learning and classification have demonstrated the strong performance with different types and levels of label noise on three synthetic IDN datasets (MNIST, CIFARIO, SVHN) and one real-world noisy dataset (CIFAR-ION) The rest of this paper is organized as follows. In Sec. 2, we review related work on label-noise learning especially modeling noisy labels and contrastive learning. In Sec. 3, we discuss how to leverage contrastive learning to learn the instant ieseant-dependent transition matrix better. In Sec. 4, we provide the empirical evaluations of the proposed method. In Sec. 5, we conclude our paper.
LABEL-NOISE LEARNING AND CONTRASTIVE LEARNING
Problem setting. Let Đ be the distribution of a noisy example (X, P) E (x.......,.,, where X denotes the variable of instances, Y the variable of noisy labels, X' the feature space, f1...., . C} the label space, and C the size of classes. In learning with noisy labels, clean labels are not available, given a noisy training sample S = {ax,, J independently drawn from D. the aim is to learn robust classifier from the sample S.
The noise transition matrix T(x)). The transition matrix T(x) has been widely used to model
label-noise generation. The ij-th entry of the transition matrix Tey (z) = P(Ý = j|Y = i = x) represents the possibility of the clean label Y = of instance flip to the noisy label Ý = j. Existed methods can learn statistically consistent classifiers when the transition matrix is given (Liu & Tao
2015 Goldberger & Ben- Ben-Re 2017: Yu et al. 2018 Xia et al. 2019 2020: Li et al.) 2021). The reason is that, the clean-class posterior P(Y can be inferred by using the transition matrix and the noisy-class posterior P() PPttrnni et al. 2017) i.e.,
T(x)[P(Y 1|x). ..... P(Y Cl2)) [P(Ý = 11x), ........................ P(ÿ = C
In general, the transition matrices are not given and need to be estimated. Without any other as- sumptions. to learn the transition matrix for an instance, its clean-label information has to be given (X al. 2019 Yang et al. 2022)
Learning the transition matrix T(x). The clean-label information is crucial for learning the transition matrix. To learn transition matrices for all instances, 1). existing methods first learn some of the transition matrices in training sample by inferring the clean-label information. 2). Then, by
2

---

Under review as a conference paper at ICLR 2023
making additional assumptions, the learned transition matrices can be used to help learn the other instance-dependent transition matrices.
Specifically, to learn some of the transition matrices in training sample, existing methods try to infer clean-label information of some instances. Then the transition matrices of these instances specific class
be learned. For example, if some instances can be identified which belong almost surely (i.e., anchor points), the transition matrices of these instances can be learned (Liu & Tao 2015). If Bayes optimal labels of some instances can be identified, their Bayes-label transition matrices can be learned (Yang et al. 2022). If clean-class posteriors are far from uniform (i.e.. sufficiently seattered), the transition matrices enclosing X( with minimum volume unique and can be learned (Li et al. 2021).
Once some of the transition matrices are learned, different assumptions have been proposed to utilize the learned transition matrices help learn transition matrices of other instances. For example, the manifold assumption, where the instances that are close in manifold distance have similar transition matrices Cheng al. (2022): ; clessddep assumption, where instances with the same clean
labels have the same transition matrices (Liu & Tao 2015 Patrini et al.) [017 Li et al. [202]);
part-dependent assumption (Xia al. 2020), where the instances with similar parts have similar transition matrices.
Contrastive learning. Contrastive learning (Sermanet et al. 2018 Dwibedi et al. 2019 Chen al. 2020aa He et al.) 2020). which could learn semantically meaningful features without human annotations (Hadsell et al. [2066] Wu et al., 20188, is an important branch of unsupervised represen- tative learning using methods related to the contrastive loss (Hadsell et al. 2006).
Existing methods have shown that semantically meaningful features are a very important character- istic in the human visual system. Humans usually use their existing knowledge of visual categories to learn about new categories of objects, where the visual categories are often encoded as high-level semantic attributes (Rosch al., 1976 Su & Jurie 2012). Contrastive learning, which helps in learning semantically meaningful features, is therefore very useful in inferring clean labels. Empir- ically, contrastive learning shows superior performance to other unsupervised-leariing techniques
on different datasets on the classification task (Chen et al. 20206 Niu et al. 2021).
In this paper, we adopt an unsupervised instance discrimination-basee representative learning ap- proach, MoCo (He et al. (222)). The basic idea of contrastive learning is that the query representa- tion should be similar its matching key representation and dissimilar to other key representations, i.e., contrastive learning can be formulated as dictionary look-up problem. Given an image æ, the corresponding images obtained by using different augmentations are Ty and Tx. The query repre- sentation generated by the backbone fa is fa fa(xz), The corresponding key representation generated by another backbone fe... fe... (xx). ) The key representation will be stored a queue. To learn the representation, for each iteration, MoCo optimize 6 according to the following loss function (He
ct al. 2020):
exp(folxg) S0mo (2x)/T) Σ log exp(So((g) Somo (xx)/T) + Ex, exp(So(Fr) Somo (2le)T) 24
Lmo
N
where 2/2 is another image different from x, femo (I is the key representation for 2/g, and r is the temperature. Then fmo updated by according the parameter 6, i.e., 6mo & pbmo + (1 1,) where μ E [0, 1) is the
momentum.
3 CONTRASTIVE LABEL-NOISE LEARNING
Motivated by the success of contrastive learning on the classification task. Previous work shows that some methods based on contrastive learning even could achieve comparable performance with supervised methods some datasets (He et al.) 2020; Chen al. 2020b). In this section, intro- duce Contrastive label-Noise Learning (CoNL), which aims to effectively leverage the advantage of contrastive learning. An overview of the method is shown in Fig. [ and described in Algo.
3

---

Under review as a conference paper at ICLR 2023
Contrastive co-selecting stage
0,,
augmentations
(P10
MIP
Pa, (Y)))
loss
Co-select
MLP
Ps.(11))
backbone
images
Ti,
Constraint T(X) revision stage
CE loss
Confident
Mw
P3(F1X)
((1))
Ty(x)
loss
fa
images
Figure 1: A working flow of our method CoNL.
3.1 LEVERAGING CONTRASTIVE LEARNING FOR CO-SELECTING
We aim to accurately select confident examples by leveraging contrastive learning. To achieve it, we utilize the contrastive method MOCO learn the visual representations on training instances. Then, the learned representations obtained by applying strong and weak data augmentations are employed to learn two classifiers and transition metrics, respectively. To select confident examples, we exploit both the estimated clean-class posterior and consistency predicted clean labels of its neighbors. The details of contrastive se-selectingg are as follows.
Firstly, produce the visual representations, backbone neural network fa :x Z is trained by only using the training instance via Eq. (T). By employing the fg. we can obtain the representations Z {2 [2])"_ )==1 and " {2," ) li=1 based on strong and weak data augmentations 4 and h" respectively, where o = Lg( ( (I:)) and z," = fz( (xx)).
Let 9o; and Jøg be two classifier heads modeling Ps, (Y and (Y with learnable param- eters Φ1 and Φ2, respectively. Let Tú, and Tc2 be two transition matrices that modeled by neural networks with lcarnable parameters Ç1 and (2, respectively. To help learn transition matrix by em- ploying the visual representations Z" and 2". We train two classifier heads 96, and Iø, and two iransition matrices Te, and Te, simultancously on Zı and Ze by minimizing the cross-entropy loss, respectively. Specifically, the objective
follows.
(d1, dz. G.,6)= arg min
Σ
(ūs log(ges (z)) is log(ge> (="") (+a))a(()))) (2)
N
$1.02.C1.5a
In this training process, the parameter of backbone fø fixed. There are several advantages. 1). By employing the representations that are independent of label errors, the classifiers can be better learned. Intuitively, previous work shows that representations learned by self-supervised learning contain the semantic information which usually correlates with clean labels (Wu et al. 2018 Niu al. 202Ip. It implies that the representations contain some information about clean labels. In the training process, the visual representations are used as inputs of the classifier head. Then the learned classifier head also contains some information about these representations and clean labels. 2). By keeping fg fixed, the learning difficulty of the transition matrix is reduced. In the learning process, only two simple models, i.c., the classifier head and the transition matrix, need to learned. Since the visual representations could contain some information about clean labels, by also employing the noisy labels, the transition matrix can be effectively estimated.
Moreover, train two classifier heads gø, and 9o, with different data augmentations. In such way, the two classifier heads are encouraged to be diverse and have different learning abilities. Then they can filter out different types of label errors and select different confident examples. We will
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Contrastive label-Noise Learning (CoNL)
Input: A noisy training sample S, noisy validation sample S. Get fg by employing MoCo and all training instances in S Minimize the Eq. 2 to learn the parameters Φ1, d2, (1, E2
Get the confident sample S S US, where Sj and Sir are generated according to Eq. and Eq. {
Select the best transition matrix To and corresponding classifier head fg by employing the
validation sample S,;
Get revised parameters 0', ø' and (" on the training sample Š and the confident sample St by employing Eq. (7) Output: fg. 9 -
illustrate our confident sample-sclection method on 93, which is trained on the representations with strong augmentations. The same sample-selection method are employed for Dō.
Specifically, the trained classifier head 9æ. is employed to relabel all instances and get a set of examples 5* {14,, Des if H To select confident examples, we employ the combination of two criteria. The basic idea is that an instance is reliable if 1). The confidence of the predicted clean label is high 2). Its corresponding predicted clean labels are consistent with its neighbor examples' predicted clean labels.
To determine whether example (æ;, 0t. Vf") should be the confident example not, the confidence of the predicted clean label and the predicted clean label consistency of z; have to be calculated. The confidence of the predicted clean labels can be directly obtained via Pă, (ara)s which is the üf-th coordinate of the output 9ä, (xz). The predicted clean label consistency ry of a; is calculated as follows.
1 K VEA!
1
where (xx) contains neighbors of the instance Ij, when the strong augmentations are applied. obtained according to the cosine similarity of features extracted from the backbone. The K* nearest neighbors with high cosine similarity are selected the neighbors of z;-
By combining two criteria together, the example (æ4, De, Ui) is considered to be an confident example if both criteria 94, (z+)gg > À and > satisfy, where À and are hyper-parameed Finally, the confident sample St selected by the trained classifier heads 9 is as follows.
S = (0,, Dc. 0r)r > T, T, (Zi) > X, vi 1,2 NN.
(4)
By applying the weak data augmentation on training instances and using the same selection method for 9ā. the confident sample S!"" selected by 9éz can also be obtained, i.e.,
S = (zx Dt. Ur)r: > T, T,9 (I:)sy A. Vi = 1.2.
N}.
(5)
Then, to utilize different confident examples obtained by applying different data augmentations, union two confident samples, i.e., S = SY US$ which will be used for constraint T(x) revision.
3.2 CONSTRAINT T(x) REVISION
We improve T-revision (Xia et al. 2019) by utilizing the confident sample S to refine the transi- tion matrix which depends on the instance. The philosophy of constraint T(x:) revision is that the favorable transition matrix would make the classification losses on both clean labels and noisy la- bels small. Once we have the confident sample S) containing both noisy labels and predicted clean labels. We could regularize and refine the transition matrix by minimizing classification losses of the
both noisy labels and predicted clean labels of the selected confident examples. After transition matrix, can also help learn better the representation and the classifier head. Therefore, in this stage, also fine-tune the representation and the classifier head.
5

---

Under review as a conference paper at ICLR 2023
Specifically. By comparing the the validation accuracy of gá, T1 (-) and 9 ()TTa (-), we select the best transition matrix and the classifier which denote as T/g and J3. respectively. Let M be the number of confident examples. Reminding that th and thd are strong and weak data augmentations, respectively. To minimize the classification loss Le on the predicted clean labels, the confident sample S₁ is employed. The loss function is as follows.
Ly (5
Σ
(ilos(gg(/s (fa(((()(())))) + (logls
(6)
M
(2.8.9)ES:
To minimize the classification loss on noisy labels, the noisy training sample $ is employed, i.e.
C (5. fo. Fo.
Σ (o (f)))
N
(w.W)s
+ūlog(g (x) ((
By combining two losses Ly and Ly together, fine-tune the transition matrix T3 the classifica- tion head Jg and the backbone model Sa by employing the objective function:
10.3'. 1.322 22 = arg min (Ly (St. fo) Lg (S TT:
,,
In Section we show that by employing constraint T(X) revision, both the estimation of the transition matrix and the classification accuracy can be dramatically improved.
EXPERIMENTS
In this section, we present the empirical results of the proposed method and state-of oot--e--attrr---- methods on synthetic and real-world noisy datasets. We also conduct ablation studies to demonstrate the effectiveness of contrastive co-selecting and constraint T(r)) revision.
4.1 EXPERIMENT SETUP
Datasets. We empirically verify the performance of our method on three synthesis datasets, i.e.. Fashion-MNIST (Xiao al., 2017). SVHN (Netzer et al.] (00IT). CIFAR-10 (Krizhevsky et al.] 2009), and one real-world dataset, i.c., CIFAR- ION (Wei et al. (022). Fashion-MNIST contains 70,000 28x28 grayscale images with 10 classes total, and 60,000 images for training and 10,000 images for testing. SVHN contains 73,257 training images and 26,032 testing images. CIFAR- 10 contains 50,000 training images and 10,000 testing images. Both SVHN and CIFAR-10 have 10 classes of images, and the image size is 32x32. The three datasets contain clean labels. We corrupted the training data manually according to the instance-dependent label noise generation repeated five times. CIFAR- is
method proposed in (Xia et al.) 2020). All experiments a real-world label-noise version of CIFAR-10, it contains auman-annotated noisy labels with five different types of noise (Worst, Aggregate, Random 1, Random 1, and Random 3). For all datasets, we leave out 10% training examples as noisy validation set.
The baselines used in our experiments for comparison using standard cross-entropy loss on noisy data directly; 2), GCE (Zhang & Sabuncu| 2018), which standard cross-entropy loss and mean absolute error to train model noisy data; 3), Mentor- Net (Jiang et al. 2018), which pretrains a model to select reliable samples for the main model: 4),
Baselines.
1). CE, training the model
Co-teaching Han al. 2018). which trains two models simultancously select reliable samples for each other; 5), Reweight (Liu & Tao 2013). which exploits importance reweighting method to estimate unbiased risk defined on clean data using noisy data; 6), Forward (Patrini et al. (017). us-
ing class-dependent transition matrix to correct loss function; 7), PTD (Xia et al.) 2020), estimating instance-dependent transition matrix through part-dependent transition matrices; 8) CausalNL (Yao)
et al. bb, which explores using causal mechanism to excavate clean-label information on noisy data; 9), MEIDTM (Cheng et al. 2022). ) which uses Lipschitz continuity to constrain the transition matrix: 10), BLTM Yang al.) 2022), which using Bayes optimal label to learn instance-dependee transition matrix; 11), NPC (Bac et al. 2022). which proposes a ppsst-proceseeee scheme to calibrate
the prediction of noise-robust classifier.

---

Under review as a conference paper at ICLR 2023
Table 1: Means and standard deviations (percentage) of classification accuracy on Fashion-MNIST.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CE
93.16 + 0.02
92.68 + 0.16
91.41 0.16
87.60 + 0.33
71.76 + 0.77
GCE
92.52 + 0.14
92.52
: 0.09
91.49 + 0.09
88.49 + 0.23
76.29 + 0.82
MentorNet
93.16 = 0.01
91.57=0.229
90.52 =
88.14
61.62 + 1.42
CoTeachiig
94.26 + 0.06
91.21+0.31
90.30 0.42
89.10
63.22 + 1.56
Reweight
93.42 + 0.16
93.12+0.18
0.18
88.51 + 1.52
75.00 + 5.28
Forward
93.48 + 0.11
92.82
91.05
87.82 1.81
78.34 + 2,98
PTD
91.55 + 2.47
5.500
75.21 + 1.81
66.21 + 5.15
CausaINL
91.63 + 0.18
0.31
90.68 0.37
90.01 0.45
78.19 + 1.01
BLTM
91.28 + 1.93
91.20 0.27
82.42 1 1.51
67.65 5.65
MEIDTM
86.00 + 0.84
80.99 0.64
73.12
2.34
63.81 3.02
58.60 + 3.32
NPC
88.78 = 0.30
88.05 - 0.02
84,99 1.20
82.59 + 1.22
70,58 + 4.43
CoNL-NR
92.14 + 0.07
91.86
0.26
91.40 0.15
90.09 + 0.41
85.84 + 0.71
CoNL
94.98 + 0.14
94.20 0.23
92.92 1.05
91.01 + 1.83
86.01 + 1.83
Table 2: Means and standard deviations (percentage) of classification accuracy on SVHN.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CE
95.75 + 0.07
95.33 + 0.18
93.74 + 0.29
91.13 + 0.32
72.43 + 2.78
GCE
95.09 + 0.03
94.37 - 0.13
86.39 :: 5.12
81.95 + 1.45
63.20 + 2.75
MentorNet
95.54 + 0.12
94.76 0.16
92.39 0.18
90.41 + 0.49
61.23 + 2.82
CoTaachiig
94.66 + 0.36
93,93 0.31
92.06
1:066000 0.31
9
67.62 1 1.99
Reweight
95.91 +: 0.44
94.23 1: 2.53
91.06 4.099
87.92
85.30 + 0.10
Forward
96.12 = 0.11
95.84
94.07
87.38
82.02 + 4.81
PTD
72.90 = 1.31
75.68 19.88+940 9.43
75.01 + 1.98
31.59 5.58
30.58 + 2.32
CausaINL
94.20 + 0.09
94.06 + 0.23
93.86 +
93.82 0.45
85.41 2.95
BLTM
93.88 0.55
92.66 1.53
92.18
84.33 5.44
76.19 5.17
MEIDTM
94.76 = 0.10
93.56 0.22
91.11 + 0.42
86.11 0.34
72.66 + 2.50
NPC
94.36 + 0.06
93.41 + 0.07
90.31
0.59
84.81 + 1.80
70.15 + 0.76
CoNL-NR
92.64 + 0.17
92.84
0.03
92.48 0.16
89.68 + 1.35
78.90 + 3.05
CoNL
96.81 + 0.09
96.29 0.47
95.37
0.88
91.73 2.53
86.51 + 1.14
Implementation. We implement our algorithm using PyTorch and conduct all our experiments on a ResNet-34 as the backbone
RTX 3090. We use ResNet-18 18 as the backbone for for SVHN and CIFAR-10. For the classifier head and transition matrix generator, we use a two layers MLP with ReLU activation function. The final layer of the transition matrix generator is initialized to generate a diagonal largest transition matrix. To learn the backbone, we follow the same settings of MoCo (He et al.) 2020). We set the temperature 7 0.2 and set μ 0.999. The size of the queue is 12800. Total epochs are 1000. When training the classifier heads and transition matrices, we use SGD with momentum 0.9, weight decay 10 batch size 128 and initial learning rate of 10- to optimize the networks. The learning rate is divided by 10 at the 5th epochs and 7th epochs. We set 10 epochs in total. When revising the transition matrix, we use Adam with an initial learning rate 10-4. The learning rate is divided by 10 at the 5th epochs and 7th epochs. We set 10 epochs in total. After that, we optimize the backbone and classifier head using SGD with momentum 0.9, weight decay 10-4, batch size 128 and an initial learning rate of 10-2 At the same time, we optimize the transition matrix generator using Adam with an initial learning rate 10-4, The learning rate is divided by 10 at the 10th epochs and 20th epochs. We set 30 epochs in total.
4.2 CLASSIFICATIONS ACTUARIES ON DIFFERENT DATASETS
We conduct experiments on Fashion-MNIST, SVHN, CIFAR-IO 10 and CIFAR-1ON. The noise rate for Fashion-MNISS SVHN and CIFAR- 10 is ranged from 0.1 to 0.5. The ones with CoNL-NR rep- resent the results of our algorithm without constraint T(X) revision. The ones with CoNL represent
the results after constraint T T(X) revision. As shown in Tab. [ Tab. 2 and Tab. 3 The proposed method is have outperform to other methods by a large margin when the noise rate is large.
We also conduct experiments on the real-world aabel-noise dataset CIFAR-1ON, The results of full types of noise (Worst, Aggregate, Random 1, Random 1, and Random 3) are shown in Tab, g The experiment results show that our method also works well on the weal-world label-noise dataset.
7

---

Under review as a conference paper at ICLR 2023
Table Means and standard deviations (percentage) of classification accuracy on CIFAR- 10.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CE
87.81 + 0.15
85.90
0.30
82.67 = 0.31
74,49 + 0.95
46.81 + 2.52
GCE
84.24 = 0.38
80.22 +: 1.30
69.31 = 0.18
56.86 + 0.92
53.44 + 1.28
MentorNet
86.87 + 0.14
83.89+0..
77.83 0.28
+1.66+0977
47.89+2.03
CeTachngg
90.06 = 0.32
87.16
81.80 0.26
63.95
45.92:12.21
Reweight
89.63 = 0.27
87.85 0.97
81.29 6.49
3.775
75.14 2.40
Forward
88.89 = 0.18
87.83
82.01 3.29
79.49
71.11 + 8.78
PTD
73.50 + 3.04
71.64 3.13
64.34 = 12.38
10.93 62.53
$1.04 + 8.28
CausaINI.
83.39 0.34
80.91 1.14
79.05 0.54
79.08 0.50
76.56 + 0.02
BLTM
80.16
77.50
1.30
71.47
2.33
63.20
48. 9.03
MHIDTM
86.52 0.38
82.93
0.44
77.35
0.21
68.21 2.09
57.84 + 3.51
NPC
84.83 0.22
0.28 83.13
79.48 0.31
73.85 0.41
67.04 + 0.06
CoNL-NR
90.04 = 0.18
90.23 0.16
89.44
0.30
87.03 + 1.41
74.14 + 6.46
CoNL
94.37 0.16
93.88 0.17
91.37 3.38
89.09 + 3.05
84.79 + 1.63
Table 4: Means and standard deviations (percentage) of classification accuracy CIFAR-1ON.
Worst
Aggregate
Random
Random 2
Random
CE
79.39
0.35
87.91 + 0.18
86.05 + 0.13
86.12 =: 0.12
86.12 = 0.16 80.89 0.82
GCE
75.45
0.31
82.77
0.13
81.18 0.22
80.39 0.54
MentorNet
77.91
0.38
7
77.12
76.03
76.57 = 0.18
CoTeaching
81.86
0.40
82.45 0.08
82.90 0.46
82.95 0.26
82.66 - 0.12
Reweight
77.68 + 2.46
1.094
88.44
88.15 0.13
88.21 = 0.28
Forward
79.27 + 1.18
89.22
86.84 0.97
87.63 = 0.39
PTD
67.72 3.33
78.20
84.19 0.51
84.36 +.36:000 0.55
69.31 = 0.43
CausalNL
73.09 2.01
0.38
80.80 0.50
81.26 0.69
81.02 = 1.48
BLTM
68.21 1.67
79.41 + 1.00
78.09 1.03
76.99
1.23
76.34 0.58
MEIDTM
79.59
0.27
87.81 + 0.52
0.18
87.86 = 0.21
NPC
75.53
1.64
84.44 0.47
81.87 0.80
81.47 0.95
82.22 + 0.58
CoNL-NR
86.52 + 0.19
89.81 + 0.31
89.63 + 0.19
89.43
0.27
89.07 = 0.33
CoNL
87.07 0.68
93.17 1 0.17
91.65 0.32
91.51 0.15
91.28 + 0.36
4.3 ABLATION STUDIES
We perform ablation studies on Rishion-MNISS SVHN and CIFARIO including the performance of contrastive co-selecting, constraint (xx) revision and the inffuence of contrastive learning on the accuracy. We leave results on Fashion-MNIST our appendix due to the limited space.
4.3. CONFIDENT EXAMPLE SELECTION
We illustrate the performance of contrastive co-selecting in Tab. 5 Tab. and Tab. 11 The results demonstrate that contrastive co-selecting can accurately select confident examples. Specifically, under 50% of instance-dependent noise, it select least 35.56% of examples with the clean ratio at least 97.88 87. Moreover, the noise rate of the selected example set S) also close to the real noise rate. This could make great contribution to the revision of transition matrix T(z).
4.3.2 THE ESTIMATION ERROR OF THE TRANSITION MATRIX
We show estimation error of transition matrix before and after constraint T(X) revision. To caculate the estimation error, compare the difference between the ground-truth transition matrix and the estimated transition matrix by employing 4 norm For each instance, we only analyze the estimation enror of specific low since the noise is generated by results are showed in Tab. 7 Tab. and Tab. [3 The results show that constraint T(X) revision can effectively reduce the estimation error of transition matrix.
of T(x). The experiment
8

---

Under review as a conference paper at ICLR 2023
Table 5: Performance of contrastive co-selecting on SVHN.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
selection ratio
37.18 0.49
37.71 : 0.16
37.25 + 0.09
36.85
0.19
35.56 +: 0.57
Noise rate
13.31 0.12
21.01 = 0.07
30.63 0.08
40.29
0.14
49.86 + 0.30
clean ratio
99.40 0.02
99.38 + 0.02
99.40 0.03
99.33 0.13
97.88
1 2.82
Table 6: Performance of contrastive co-selecting on CIFAR-I0.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
selection ratio
66.33 2.33
69.06 + 0.84
63.54
0.47
57.70
0.99
47.90 + 0.83
Noise rate
13.64 0.09
21.37 + 0.06
31.49 + 2.10
40.53 - 0.22
49.73 1 0.12
clean ratio
99.14 + 0.04
99.15 + 0.05
96.89 4.83
98.45 = 1.95
99.18 + 0.36
Table 7: Transition matrix estimation error on SVHN.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CoNL tw/o MoCo)
0.193 +: 0.008
0.204 0.020
0.229 0.026
0.251 +: 0.001
0.285 + 0.014
CoNL-NR
0,256 + 0.061
0.193
0.006
0.214
0.006
0.240 + 0.008
0.305 = 0.015
CONI.
0.177 0.020
0.188 0.005
0.206
0.009
0.226
0.011
0.231
0.019
Table Transition matrix estimation error on CIFAR-10.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CONL (w/ MoCo)
0.217 # 0.006
0.221 + 0.008
0.253 + 0.003
0.303 + 0.001
0.384 = 0.003
CoNL-NR
0.206 + 0.011
0.198
0.001
0.252 + 0.029
0.285 + 0.022
0.350 + 0.018
CoNL
0.168 1 0.009
0.195 1 0.003
0.229 1: 0.019
0.255 1 0.009
0.262
0.005
Table 9: The test accuracy of with MoCo and without MoCo on SVHN.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CONL. (w/o MoCo)
94.88 + 0.13
94.56
0.12
93.85
0.24
86.47 + 1.17
82.75
2.33
CoNL-NR
92.64 + 0.17
92.84
0.03
92.48
+: 0.16
89.68 + 1.35
78.90 +: 3.05
CoNL
96.81 1 0.09
96.29
0.47
95.37
1 0.88
91.73 1: 2.53
86.51 i 1.14
Table 10: The test accuracy of with MoCo and without MoCo on CIFAR-10.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CONL (w/o MoCo)
83.88 + 0.17
82.53
0.46
80.51
0.22
75.34 + 0.62
68.62 3.56
CoNL-NR
90.04 1 0.18
90.23
0.16
89.44 1 0.30
87.03 1: 1.41
14.14 6.46
CoNL
94.37 1 0.16
93.88
0.17
91.37 3.38
89.09 1: 3.05
84.79 1.63
4.3.3 ACCURACIES WITH OR WITHOUT EMPLOYING CONTRASTIVE LEARNING
We conduct the experiments with without employing contrastive learning on different datasets. For CoNL. (w/o MoCo), the backbone method fo is not trained by employing MoCo, but during the co-selecting stage, we do not fix its parameter. The other settings are same as CoNL. The experiment results are showed in Tab, 9 Tab. and Tab. [2] The empirical results clearly show that the contrastive learning technique dramatically improves the robustness of the learning model and is powerful for inferring the clean-label information.
5 CONCLUSION
Since both instances and noise labels are available, the main difficulty learning instance-dependent transition matrices is due to the lack of clean-label information. Motivated by the great success of self-supervised learning in inferring clean labels. In this article, we propose CoNL (Contrastive Label-Noise Learning). which effectively utilize self-supervised learning to learn instance- dependent transition matrices. Empirically, the proposed method have achieved state-of the per- formance on different datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
HeeSun Bac, Seungjae Shin, Byeonghu Na, JoonHo Jang, Kyungwoo Song, and II-Chul Moon. From noisy prediction true label: Noisy prediction calibration via generative model. In Inter- national Conference on Machine Learning, pp. 1277-1297. PMLR, 2022.
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. arXiv preprint arXiv:2106. 15853, 34:24392-24403. 2021.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learn- ing. pp. 1062-1070. PMLR, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04297 2020b.
De Cheng, Tongliang Liu, Yixiong Ning, Nannan Wang, Bo Han, Gang Niu, Xinbo Gao, and Masashi Sugiyama. Instance- label-noise learning with manifold-regullrizeer transi- tion matrix estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16630-16639, 2022.
Madalina Ciortan, Romain Dupuis, and Thomas Peel. A framework using contrastive learning for classification with noisy labels. Data, 6(6)611, 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Tem- poral
learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1801-1810, 2019.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR'OL 06), volume 2, pp. 1735-1742. IEEE, 2006.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, volume 31, pp. 8527-8537, 2018
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data- driven currieulum for very deep neural networks corrupted labels. In ICML, pp. 2309-2318, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2019.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint arXiv:2009.07991 2020.
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label- noise learning without anchor points. arXiv preprint arXiv: 102 02400, 2021.
10

---

Under review as a conference paper at ICLR 2023
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447 4411 2015.
Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. In Interna- tional Conference on Machine Learning, pp.
PMLR, 2021.
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y.Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self- ensembling, In ICLR, 2019.
Kento Nishi, Yi Ding, Alex Rich, and Tobias Hollerer. Augmentation strategies for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 8022 4022-8031, 8031, 2021.
Chuang Niu, Hongming Shan, and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. arXiv preprint arXiv:2103.09382, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pp. 1942-1952, 2017.
Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braem Basic objects in natural categories. Cognitive psychology, 8(3):382-439, 1976.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Yu Su and Frédéric Jurie. Improving image classification using semantic attributes. International journal of computer vision, 100(1):59-77, 2012
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-ferst AAAI con- ference artificial intelligence, 2017.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13726-13733 2020.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: study using real-world human annotations. In International Confer- ence on Learning Representations, 2022. URL htt.l::t////// /ttp::s////ere//////// wwwreeeiiww....... net. /eee//forree forum? id TBWA6PLJZ0C
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurIPS, volume 32, pp. 6835-6846, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. NerurIPS, 33:7597-7610. 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: (o0X:v/700.0074:: 2017.
11

---

Under review as a conference paper at ICLR 2023
Shuo Yang, Erkun Yang. Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependers bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning, pp. 2230222531... PMLR, 2022.
Yazhou Yao, Zeren Sun, Chuanyi Zhang, Fumin Shen, Qi Wu, Jian Zhang. and Zhenmin Tang. Jo-src: A contrastive approach for combating noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5192-5201, 2021a
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error for transition matrix in label-noise learning. In NeurlPS, volume 33, pp. 7260-7271, 2020.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under a structural causal model. volume 34, pp. 4409 4420, 2021b.
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In ECCV, pp. 68 68-83, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurlPS, pp. 8778-8785 2018.
A ABLATION STUDIES ON FASHION-MNIST
We demonstrate results of ablation studies Fashion- including the performance of con- trastive co-selecting, constraint Txx) revision and the influence contrastive learning on the aceu- racy in Tab. 11 Tab. 1 and Tab. 12 respectively.
Table 11: Performance of contrastive co-selecting Pashion-MNIST
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
selection ratio
23.28 0.84
26.00 + 1.09
23.46 1.10
22.37
1.42
11,94 4 1.87
Noise rate
11.82 0.20
20.18 = 0.13
30.12
0.21
39.47
0.26
49.21 1 0.42
clean ratio
99.76 + 0.11
99.81 + 0.06
99.88
0.03
99.87 + 0.06
99.74 + 0.32
Table 12: Test accuracy of with MoCo and without MoCo on Rashion-MNIST
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CoNL (wlo MoCo)
91.79 = 0.15
90.76
0.14
87.46
2.32
84.43 + 1.43
75.16 5.13
CoNL-NR
92.14 + 0.07
91.86
0.26
91.40 + 0.15
90.09 + 0.41
85.84 0.71
CONI.
94.98 + 0.14
94.20 0.23
92.92
1.05
91.01 + 1.83
86.01
1.83
Table 13: Transition matrix estimation error on Rishion-MNIST
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CoNL (wo MoCo)
0.247 1: 0.005
0,360 1 0.012
0.450 + 0.011
0.582 1 0.023
0.674 : 0.020
CoNL-NR
0.236 + 0.003
0.327 + 0.021
0.409 + 0.010
0.528 + 0.013
0.674 = 0.018
CoNL.
0,238 + 0.006
0.324
0.005
0.391
0.008
0.467 + 0.013
0.540 = 0.003
B EXPERIMENTS ON 100 AND WEBVISION
To verify whether the proposed method can still work well when the number of class increase, we conduct the experiments on CIFAR- 100 and Web datasets. Specifically, for CIFAR-100, we keep the same experiment setting as 100. For WebVision, we trained a standard ResNet-18 and an inception-resned v2 (Szegedy et al. 2017) on Vision for 1000 epochs using MoCo v2 to obtain the pretrained model. We follow the previous work (Chen et al. 2019) to select the first 50 classes of the Google image subset as the training set and leave out 10% of training examples as a noisy validation set. Then train the model with the proposed CoNL., Other experiment settings
12

---

Under review as a conference paper at ICLR 2023
same the experiment on CIFAR-10. 10. We test the model on the human annotated Web Vision validation set. The test accuracy of CoNL with ResNet backbone is 64.88% and the test accuracy of CoNL with inception-resnet v2 backbone is 70.80%. The experiment results of CIFAR- 100 are shown in Tab. 14
Table 14: Test accuracy of CoNL on CIFAR-100.
IDN-10%
IDN-20%
IDN-30%
IDN-40%
IDN-50%
CoNL-NR
40.78 1.07
39.94 1.51
38.30 1.77
36.25
1.69
32.25
0.85
CoNL
74.13 + 0.34
72.15 + 0.52
69.96 = 0.71
65.40 + 2.76
59.09 = 1.78
C DIFFERENCES BETWEEN CONTRASTIVE CO-SELECTING AND PREVIOUS WORK
Previous work Co-Teaching (Han al. 2018 learn two classifiers to select confident examples for each other, and filter eπrors from the biased selection in the first mini-batch. In our work, two classifiers are only used the co-selection stage and do provide supervised signals for each other. We aim to design method that can select examples as many as possible. Previous work AugDese (Nishi et al. 2021) explores how to use data augmentation techniques for different permu- tations and combinations improve the generalization and robustness of models without impacting the memorization effect negatively, i.e., weak augmentation techniques for pseudo labels generation and strong augmentation techniques for the mack-propagation step update model's 'ss parameters. In this paper, different data augmentation strategies are used to enable contrastive co-selecting to select more reliable examples under different noise rates.
13


Reviews:
[{"review_id": "yk5tK70WcSL", "reviewer": "Reviewer_RfYS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for estimating instance-dependent transition matrix using contrastive pretraining. The method starts from the pretrained network using contrastive loss, and train a set of classifiers to divide data into confident set. They are further used to refine classifier, transition matrix, as well as the backbone. In experiments, the proposed method is tested on standard noisy-label learning benchmarks, demonstrating improved performance over existing works.", "summary_of_the_review": "The use of pretrained network for image classification is well established, so it is not surprising if it improves when applied to label noise transition matrix estimation problem. Unfortunately, the study is not well conducted and it is hard to understand which part of the proposed framework contributed the most to the empirical success. Additional study to highlight the effectiveness of transition matrix learning should be provided.", "strength_and_weaknesses": "* Strength\n  - The paper tackles important problem of transition matrix estimation using self-supervised pretraining.\n\n* Weakness\n  - The paper is not well written. \n  - Overall, the `contrastive` is used only to provide pretrained backbone, so it is unclear why the method is to be called a contrastive noisy-label learning. In theory, the pretrained backbone can be replaced with any networks.\n  - How much performance improvement is attributed to the use of pretrained model and the use of fine-tuning? While the use of contrastive pretraining is useful in practice, it does not add much scientific value to the community. For example, a fair comparison to previous works would be to employ the same pretrained backbone and evaluate their methods, as some previous works are agnostic to the network initialization (e.g., BLTM, NPC).\n  - How does method compared against more recent works, such as [DivideAndMix](https://arxiv.org/pdf/2002.07394.pdf)?\n  - In Equation (2), are there any constraints that makes T's a transition matrix? It is unclear what it means by the transition matrix (e.g., what's the difference between transition matrix and a matrix?) and how it is regularized to learn a `proper` (which is not defined) transition matrix.\n  - In section 3.2., the use of validation accuracy in selecting the best transition matrix is problematic. How large is the validation set? Authors also need to provide ablation w.r.t. the size of validation set and its impact on the final performance.\n  - In experiment, it is unclear how `CONL-NR` model is trained. Does it mean the model is trained with the loss in Equation (7) but without updating the parameters w.r.t. transition matrix?\n\n* Misc\n  - The paper requires significant revision as there are numerous grammatical errors, inconsistent notations, etc. To list a few:\n    - There are many broken sentences, e.g., pg 3, `motivated by the success of ... classification task.`; pg 5, `Once we have the confident sample, ..., clean labels.`; \n    - Equation (6), $\\tilde{y}$ is not used.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "eZtyz0BEQVy", "reviewer": "Reviewer_SKuM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to leverage self-supervised learning features to improve the estimation of the transition matrix. The paper developed a framework for learning the instance-dependent transition matrix. The framework is composed of confident examples selecting stage using contrastive learning and constraint transition matrix revision stage. The contrastive co-selecting stage leverages different degrees of data augmentations to acquire representations with semantic information correlating with clean labels. By jointly optimizing noisy training sample loss and clean confident sample loss, Constraint transition matrix revision improve the estimation of the transition matrix and classification accuracy.", "summary_of_the_review": "This paper developed a novel framework for learning the instance-dependent transition matrix by utilizing a self-supervised learning. The idea is novel and interesting, while more clarifications in the novelty in the approaches, the comparison experiments and experiments setup. The accuracy of the experimental results should be more convincing.", "strength_and_weaknesses": "Strength: \n\n1. This paper provides a novel understanding of how self-supervised learning representation can help to construct a model with noisy label data. It is an interesting discovery that SSL representation can help learning with noisy labels. The idea that contrastive learning representations contain some information about clean labels is intuitive and novel.\n\n2. This paper also provides a novel understanding of how to jointly optimize loss function on both clean labels and noisy labels to helps estimate transition. \n\nWeakness: \n\n1. The co-selecting stage is similar to a combination of the ensemble learning method and pseudo-label generation. It is unclear why training two classifier heads with different data augmentations instead of different contrastive models, which may help decrease model-related errors?\n\n2. The authors claim that the method can achieve state-of-the-art performance on different datasets, but more datasets should be introduced, such as CIFAR-100N and WebVision to show if the methods still work when the number of classes and the transition matrix become larger.\n\n3. The experiment results of baselines are significantly different from those reported in other papers (see below). It would be better to provide the codes and models used in the paper or point out how the experiment setting can be different from previous works.\n\nXia X, Liu T, Han B, et al. Part-dependent label noise: Towards instance-dependent label noise[J]. Advances in Neural Information Processing Systems, 2020, 33: 7597-7610.\n\nYang S, Yang E, Han B, et al. Estimating instance-dependent bayes-label transition matrix using a deep neural network[C]//International Conference on Machine Learning. PMLR, 2022: 25302-25312.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WAL17_H9u9", "reviewer": "Reviewer_36DZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A novel self-supervised learning framework is proposed for noisy labels and can effectively improve the accuracy of instance-dependent transition matrices. Specifically, this method mainly contributes to two parts. First, a contrastive co-selection method is proposed to select samples with high confidence. This method obtains the basic feature extraction backbone network learned by MOCO, and then design criterions to select confident samples. Secondly, a constraint T-revision method is proposed, which uses confidence samples to fine-tune T(x) to improve the classification accuracy.", "summary_of_the_review": "This paper achieves a large performance improvement through the proposed contrastive co-selecting and constraint T (x) revision methods. We think this paper has certain innovation. However, the core innovation points of the article and the description of the title abstract are not appropriate. In the experimental part, the method will be more convincing if it can be verified with other methods (DivideMix, etc.) on large-scale real-world datasets (Clothing1M and WebVision, etc.).", "strength_and_weaknesses": "In this paper, self-supervision is used to improve the classification performance of noisy-label samples. And mining the information of the confident samples to improve the performance of the model. The method is simple and effective.\n\n\n1. Use self-supervision to improve the estimation of the transition matrix as described in the title of this paper. But the core idea of this paper is not the design of self-supervised algorithms. The self-supervised MOCO algorithm is only a pre-training method as the first step of the method, and other pre-training methods can also be used in this part. In addition, although this algorithm trains two classifiers using the data with strong and weak augmentation transformation. However, these two branches use noisy label training, and the two-branch training process does not have the idea of self-supervision. Therefore, I do not think that the algorithm is improved by self-supervision as described in the title.\n2. The innovation of this paper is weak. The idea of training two classifiers to be complementary has been extensively studied in papers from earlier years. The designed contrastive co-selecting method utilizes two criterions for sample selection, which is not very novel. At the same time, the improvement of constraint T (x) revision only uses the confident samples based on the T-revision algorithm. The algorithm has less theoretical support.\n3. The abstract part does not clearly show the specific method. It is necessary to outline the innovative idea of the algorithm in the abstract, so that readers can quickly understand the core idea of the paper. The summary part has the same problem and needs to explain the advantages of the method, the direction of future improvement, etc.\n4. Figure 1 of this paper describes the workflow of the method. However, the figure is relatively rough, which does not allow readers to quickly understand the modules of the algorithm and the specific execution processes.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6MEInQXDnU", "reviewer": "Reviewer_6eSM", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel method for estimating the instance-dependent transition matrix. The self-supervised learning are employed to mine the clean samples. And then refines the transition matrix estimated by T-Revision via employing the selected confident examples. The empirical results varifies the effectiveness of proposed method.", "summary_of_the_review": "This work proposes an interesting perspective to combine self-supervised learning and instance-dependent transition matrix.  While the mechanism of learning transition matrix by noisy labels is not clear. The empirical results can not support the  effectiveness of proposed method, especically some additional techniques, which potentially bring much performance increase. Overall, the writting is poor, and the noverty and effectiveness of proposed method need to further clarify.", "strength_and_weaknesses": "This paper refines the learned transition matrix by employing the selected confident examples, which is interesting. However, there exist some concerns as follows.\n- The idea is novel, while the formulation is not rigorous. Specifically, why use self-supervised learning to select clean samples? What is the difference with other sample selection methods. \n- The self-supervised learning has help improve the label noise learning, e.g., C2D [1], Sel-CL [2]. It is fair to epuip compared methods with contrastive learning. It can be seen that CoNL (w/o MoCo) in table 9 & 10 evidently underforms compared methods. From this viewpoint, I doubt the effectiveness of proposed method, since the performance increase completely stem from self-supervised learning.\n\n[1] Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, AlexMBronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1657–1667, 2022.\n\n[2] Li S, Xia X, Ge S, et al. Selective-supervised contrastive learning with noisy labels[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 316-325.\n- Also, the method uses different data augmentation startegies. It has been found that using two separate pools of augmentation operations for two separate tasks is beneficial for noisy label learning in [3]. I doubt that the performance increase mainly dues to such learning manner, rather than the proposed method. \n\n[3] Nishi K, Ding Y, Rich A, et al. Augmentation strategies for learning with noisy labels[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 8022-8031.\n\n- Eq.(7) is not proper for transition matrix. The transition matrix is used for the whole training dataset, rather than just only noisy datasets. This deviates the classifier consistent theory of  transition matrix.\n\n- The method only ran experiments on several simple datasets. The datasets with more classes, more complex noise structures, and real-world datasets, e.g., WebVision, are necessary to demostrate the effectiveness of proposed method.\n\n- How to guarteen that $T_{\\zeta}$ does not overfit the noisy labels? And the network structure of transition matrix is not clear. How to guarteen that network can estimate more classes?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject