Set-Level Self-Supervised Learning from Noisily-Labeled Data
Authors: Chia-Ching Lin
Shang-Fu Chen
Fu-En Yang
Yu-Chiang Frank Wang
Chin-Laung Lei
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self-supervised learning
Noisy label learning
Meta-learning
EM algorithm

Abstract: Noisy labels are inevitably presented in real-world datasets due to labeling error or visual content ambiguity. Existing methods generally approach the task of noisy label learning (NLL) by either properly regularizing the model, or reweighting clean/noisy labeled samples. While self-supervised learning (SSL) has been applied to pre-train deep neural networks without label supervision, downstream tasks like image classification still require clean labeled data. And, most SSL strategies are performed at the instance level, regardless of the correctness of its label. In this paper, we propose set-level self-supervised learning (SLSSL), which performs SSL at mini-batch levels with observed noisy labels. By corrupting the labels of each training mini-batch, our SLSSL enforces the model to exhibit sufficient robustness. Moreover, the proposed SLSSL can also be utilized for sample reweighting technique. As a result, the proposed learning scheme can be applied as an expectation-maximization (EM) algorithm during model training. Extensive experiments on synthetic and real-world noisy label data confirm the effectiveness of our framework.

Under review as a conference paper at ICLR 2023
SET-LEVEL SELF-SUPERVISER LEARNING FROM NOISILY-LABELED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Noisy labels inevitably presented in real-world datasets due to labeling er- ror or visual content ambiguity. Existing methods generally approach the task of noisy label learning (NLL) by either properly regularizing the model, reweight- ing clean/noisy labeled samples. While self-supervised learning (SSL) has been applied to pre-train deep neural networks without label supervision, downstream tasks like image classification still require clean labeled data. And, most SSL strategies are performed at the instance level, without accessing label. this paper, we propose set-level self-supervised learning (SLSSL), which per- forms SSL at mini-batch levels with observed noisy labels. By corrupting the labels of each training mini-batch our SLSSL enforces the model to exhibit suf- ficient robustness. Moreover, the proposed SLSSL can also be utilized for sample reweighting technique. As a result, the proposed learning scheme can be applied as an intitttiionnmainzzatiion (EM) algorithm during model training. Extensive experiments on synthetic and real-world noisy label data confirm the effectiveness of our framework.
INTRODUCTION
Deep learning has shown tremendous success in numerous computer vision and machine learning tasks. However, collecting a large amount of precisely labeled data for training deep neural net- work (DNN) typically time-consuming and labor-intensive. Moreover, in practice, real-world datasets are usually annotated with noisy labels. In order to alleviate possible overfitting prob- lems (Arpit et al. 2017]] ZZhang et al. (017), noisy-label learning (NI.L) has attracted the attention from researchers in related fields (Frenay & Verleysen| 2014 Song al. 2021
Recent deep-lea based NLL approaches can be categorized into two groups (Song et al. 2021 Karim al. (2022). The first group focuses on loss correction (Patrini et al. (017,eendyyck 2018 Xia et al. 2019 Wang et al. 2020 Yao et al.) 2020), which learns a wise noise transi- fion matrix to counteract the noise effect during trainin. that the predicted labels can be updated accordingly. The second group of NLL works present various sample selection algorithms (Li et al.) 2020 Nishi et al., 2021: Karim et al., 2022), aiming filtering noisy samples. Once the noisy labels are removed, semi semi-supervised learning techniques can be applied for training learning mod- els. While promising performances have been reported, the above learning strategies rely on the prediction of the derived noise transition matrix or instance weights, which require proper learning and estimation using the training data and their noisy labels.
al.
Instead of directly utilizing the noisy labels, self-supervised learning (SSL) has been recently ap- plied to NLL tasks (Hendrycks et al. 2019 Ghosh & Lan 202T Yao et al. 2021 Ortego et al., 2021). By properly designing pretext tasks, additional supervisory signal can be derived to improve robustness of the model against label noise. However, existing SSI approaches design pretext tasks by manipulating samples at the instance level, regardless of the correctness of its label. As for instance-level pretext tasks (c.g., rotation prediction or contrastive- based instance discrimination), they expected to produce proper representations from unlabeled data; it is not clear whether such techniques would result in robust representations when tackling the problem of noisy label learning (NLL).
In this paper, we propose a novel unique SSL approach for NLL. More precisely, we present a set- level self-supervised learning (SLSSL) strategy for training NLL models. As illustrated in Fig. 1

---

Under review as a conference paper at ICLR 2023
Lestance-Level SSL
Set-Level SSL
lenture similarity
prediction consistency
preductor
dop-gead
curoder
momentum eacoder
learning model
EMA model
Aegnemrttio
Angrerration
Deigrat
%
72
image
Saini-halch33
Figure 1: Illustaaionn of our set-level self-supervised learning (SLSSL). Unlike instance-level SSL approaches, our SLSSL augments an image set (e.g., mini-batch) by manipulating its labels. By maximizing the agreement between the two augmented versions, our SLSSL results in learning models which are robust to noisily-labeled data.
given a set (mini-batch) of training samples, our SLSSL augments noisly labeled data by corrupting portion of its labels for updating the DNN through single-step optimization, while the updated model is enforced to maximize the performance agreement between different augmentation versions. As detailed later in Sect. 3, our SLSSL objective is formed to estimate the class-wise noise transition matrix, allowing us to enhance the robustness of the learned model. In addition, we show that the proposed SL.SSL can utilized reweight samples for sample selection purpose. Unlike existing works that perform sample selection by assuming that instances with small losses are with clean labels, our SLSSL learns to assign larger weights to those resulting in significant performance degradation during label corruption, identifying the data with clean labels accordingly. Finally, demonstrate that our SLSSL can realized an expectation-maxiiizaiinn with E-steps focusing samples for training. As verified in our experiments, this alternating training strategy further boosts the performance of our framework.
(EM) like algorithm, training model with noisy labeled data, and M- M-steps identifying clean data
The contributions of this paper are highlighted below:
We propose set-level self-supervised learning (SLSSL) to tackle noisy-label learning (NLL) tasks, which augment image sets and enforce the model to be robust noisy labels.
By systematically corrupting the labels during training and enforcing prediction consis- tency between associated models, our SLSSL can be applied to estimate the noise transition matrix, which introduces sufficient robustness the learned model against noisy labels.
Our SLSSL can further utilized to identify the label quality each training sample, and thus sample selection for NLL can be performed accordingly.
Our proposed learning strategy be further viewed as EM-like algorithm, which al- ternates between model training and sample reweighting for improved NLL.
2 RELATED WORKS
Loss Correction for NLL A number of NLL works (Goldberger & Ben- 2017 Patrini
2017 Hendrycks et al., 2018 Xia et al.) 2019 Wang et al. 2020 Yao et al., 2020 Zhu et al. 2022) focus on estimating the class-wise noise transition matrix of noisy training data, which describes the relationships between noisy labels and their ground-truth ones and thus can be applied to refine the predicted outputs accordingly. It is shown in (Patrini et al. 2017) that minimizing such corrected loss toward noisy labels equivalent to optimizing the DNN toward the ground- truth labels. However, despite of their theoretical foundation, how to accurately estimate the noise transition matrix remains a challenging problem, especially when no clean training/validation sets available.
2

---

Under review as a conference paper at ICLR 2023
batch 5
sample
Imput
predict
Training
prediction consistency
label computtin
EMA
4ay
batch 5"
Imput
update
0'
predict
Figure 2:
Illustration our set-level self--upprrvied learning (SLSSL). Given a training mini-batch S, we first augment it by corrupting its labels and then obtain the augmented model parameter 0', We then enforce the consistency between the outputs from [(; 0*) and that from the weight-averaged model f(;;0" to encourage model robustness against label noise.
Sample Selection for NLL Selecting clean training samples according to their label confidences another alternative for solving NLL tasks. In co-training based approaches (Malach & Shalev-
Shwartz 2017] Jiang al., 2018; Han et al. 2018 Yu et al., 2019; Wei et al. 2020), two DNNs are trained in collaborative manner, with each model being trained using clean samples selected
by its peer model. More recently, (Li al. 2020 Nishi al. 2021 Karim et al. 2022 Wang let al. 2021) treat the selected clean and noisy samples as labeled and unlabeled data, respectively. They then leverage state-of-the-arr semi-supervised learning techniques for training the learning models. While achieving promising results, most of the above methods perform sample selection based the assumption that samples with small classification losses with high probabilities for the assigned labels. This might not always holds for hard samples with correct samples to be recognized.
Self-Supervised Learning for NLL Self-supervised learning (SSL) has been recently considered for approaching NLL tasks (Hendrycks et al., 2019) Ghosh & Lan 2021 Yao et al.] 2021 Ortego al.) 2021). Most existing works adapt pretext tasks that are designed for SSL on unlabeled data, such as predicting image rotations or contrastive-based instance discrimination, without utilizing the structural information contained in noisy labels for advanced supervipery signals for NLL. Recently, (Li et al., 2019) propose Meta- Meta-Learning based Noise- Tollerant (MLNT) training, in which the noisy labels are further corrupted multiple times, and a consistency constraint is designed to regularize the model toward robustness under such label corruptions. This consistency constraint can be regarded as self-super signal that takes noisy labels into account. However, as MLNT chooses to corrupt labels randomly, the pretext task is limited to model regularization, not able to identify data with noisy labels as sample selection methods do. As introduced in the next section, proposed strategy can be applied for improving model robustness and estimating label confi- dence (i.e., sample reweighting). and our experimental results would verify the effectiveness and applicability of our approach.
PROPOSED METHOD
3.1 PROBLEM DEFINITION
Suppose that image space X, and yn. E {1, ..., C} as the associated (noisy) class label, which may not nec- essarily consistent with its ground-truth label yn. Let f(:00 (): denote the NLL model parameterized by 8. Our goal is to derive robust NLL model 8", which is derived by self-ensemble exponential moving average (EMA): H + a0" + (1 ai),, with the hyperparameter controlling the update speed. With model 0* obtained, classification of test data can be performed accordingly.
given training set D = (ann Bn)), with x,, denoting the n-th image from the
using
3.2 SET-LEVEL SULEVIIPERVISED LEARNING (SLSSL)
Given a set of training samples (i.e., a mini-batch) denoted by S = {(In, N with size N, propose augment it by randomly selecting two classes # j from 21,2. C), followed by relabeling all the samples from label i using label j, as illustrated in Fig. This label-corrupted mini-batch is thus denoted by S ((xn: 1%.)) ) 1 where
3

---

Under review as a conference paper at ICLR 2023
noise transition matrix ?
Cesu
Training Set
SUSSL
In
sample
batch
Lin.
Input
0
predict
a adjusted prediction
original prediction
Figure 3: SLSSL for model training. Given a noisy dataset D, we apply SLSSL with self-supervised consistency loss LSSL for learning robust models. The observed consistency statisties are further translated into the noise transition matrix, which refines classification prediciton accordingly.
= [J. 8n - Vin otherwise.
yn. i;
(1)
We apply each augmented S to update the model by single-step gradient descent using the cross-entropy classification loss. That is, we obtain 6' by:
N te (f (xni .),
0' V Σ
where y) represents the learning rate used in such temporary updates, and tc denotes the cross entropy loss.
By repeating the above procedure M times, obtain M different models from different label corruptions of the same mini-batch, i.e., we observe {0'M each corresponding to a specific label corruption between labels (i.j)- This set of augmented models (0r, {0m m=1 would guide the learming of 0 and 8 in terms of both model training and sample reweighting, as we detail in Sect, 3.3 and Sect. 3.4 respectively.
3.3 SET-LEVEL. SSL FOR MODEL. TRAINING
Enforcing model robustness against label corruption To make our NLL model 0 (and hence 8*) robust against noisy labeling, we enforce each augmented model ) to give consistent prediction outputs the same input In. which should be close to the output predicted by 0". As depicted in Fig. 2 we derive each 6', by updating 0 on a mini-batch with two classes and being corrupted (say, represents bird and j represents airplane), the predictions of f(=;0(,) for the remaining classes (e.g., cars) are not expected to be altered by this label corruption. In other words, prediction outputs for classes other than i and are expected to be invariant to the above corruption. To achieve this objective, we modify the widely-used Dallback-Leibler (KL) divergence to define our SSL objective follows:
N D D
LSSL
Σ
Σ
f fæn; 6mm))
MN
where D represents the KI divergence computed by excluding the class dimensions and j. which are corrupted for obtaining 0' i.e.,
('z(((rn;)")) f(xn; Km)) = Σ (ot (m)))
(4)
cft.
By optimizing the SSL loss defined in equation 3 the models f(: and f(:;0") would be encour- aged to be robust to label corruption, and hence the learned model would be expected to generalize test set data.

---

Under review as a conference paper at ICLR 2023
Estimating noise transition matrix for NLL While the above SSL strategy can be viewed
model regularization technique, previous works like (Patrini et al. 2017: Hendrycks et al.) 2018 Xia
et al. 2019; Wang al. 2020; Yao et al. 2020) further deal with NLL by estimating class- wise
C-by- noise transition matrix for refining the classificaiton outputs.
To be more specific, this transition matrix specifies the relationships between noisy labels {ūn} } and ground-truth labels {yn), with the (i, entry defined by Tú = P(ún = jlyn. = i), i.e., the probability of sample with ground-truth label being mislabelled to label j. By estimating 7 and multiplying it with the class prediction output, losses calculated for classes with corrupted labels will be suppressed. In other words, by introducing such a transition matrix in to NLL models, the final prediction output will be encouraged to be aligned with the ground-trute label.
Unfortunately, estimating the above noise transition matrix T is challenging task, especially when no clean crining/validation sets are available (as the setting of our work). We now explain how we solve this task using the above set-level SSL strategy. Suppose that 6", is derived by updating on mini-batch with classes (i, being corrupted, as described in Sect. 3.2 Intuitively, if the samples with true class in the training dataset have a larger portion being mislabeled as I (say, i represents bird and j denotes airplane), the model (and hence 0*) trained on this dataset would have biased performance towards these two categories. Thus, the performance of 8', would be less sensitive to label corruption between these two classes. On the other hand, for the case where i and j are less likely to be confused (say, i represents bird and j represents truck), the associated 6' would be expected to exhibit larger performance deviation.
With the aforementioned observation and property, we propose to estimate T by measuring the deviation between On, and 0* Specifically, adapt the standard KL divergence as:
DKL(f(Zn:0") II f(zni 6m))) (olcaa (.)).
(5)
To estimate the ji,j entry Tif we collect all On that are derived by comrupting (i, across multiple mini-batches, and compute the ùnverse of the average KL divergence as:
Qu Bos (f f(zni (m)))mm)))"" Omm))²
(6)
where Egg indicates that the average computed over all 6t, 's derived by corrupting (i,j), and is a sharpening parameter. We then normalize Qi.j to obtain T; Qi.i/ Σe Qi,--
From the above objective and derivation, we are able to estimate T in a per-epoch basis. We can update the matrix via EMA: f. denotes the matrix estimated within each epoch, and we update it by T of + (1 - B)Σe. with 3 e [0, 10 controlling the update speed. As illustrated in Fig, 3 follow forward loss correction methods (Patrini al. 2017; Wang al.) 2020) and transform the model prediction output f (xx;;) 0) by multiplying it with T. This transformed prediction thus is applied for calculating the cross-entropy loss with respect to the noisy labels { And, the corrected version of cross-entropy loss then be expressed follows:
N
LCE
In log(T f(rn; 0)),
N
R=1
where the superscript T means matrix transpose. Finally, the total loss for model training is derived by combining LSSL in equation B and Loe in equation
3.4 SET-LEVEL SSL FOR SAMPLE REWEIGHTING
As discussed in Sect. another group of NLL works choose to reweight training samples based on their labeling confidence for training the learning model. We now explain how our SLSSL can also be utilized for sample reweighting well.
5

---

Under review as a conference paper at ICLR 2023
(A M
8 nR, Sar A)
sample
input
update
predict
Training
(w))
DEL Σ D
Drc
predict
label comruption
larger deviation (Dir. > Pxx) increase for @ and
batch 5"
(Nem Far Wel
8
194 En On
Dic
input
update
('(w)
predict
smaller deviation (Dka, (xx)
decrease Wi, for and
Figure 4: SLSSLL for sample reweighting. Given mini-batch S, we derive (W) and (w) using S and its label-corrupted version S", respectively. Note that large deviation between and 0' ('(w) would be observed, if such corruption is applied clean data (when comparing to deviation between 0 and 0% (w)). This learning strategy allows to increase the weights for clean training samples (and vice versa). See Sect. 3.4 for details.
As illustrated in Fig. suppose that we are given a mini-batch of training samples S {IIn In. wn) ) where Wi, denotes the weight of the n-th sample (In, Vn). For sample reweight- ing, we keep the model parameter fixed and only update the sample weights {w.,). Recall that the estimated noise transition matrix π can applied for transforming the model prediction in NLL. Intuitively, I' should be close to an identity matrix if most of the samples are correctly labeled (that is, correctly-labele samples are assigned with larger weights than those of noisily-labeled ones). In other words, the optimal weights in S should make Tiy close 0 for i * j (i.e., the off-diagonal entries) and for i = (i.e., the diagonal entries).
To achieve this, first S estimate the noise transition matrix described in Sect. 3.3 with } included in the estimation process, and then optimize [wa} based on the estimation results. Specifically, follow the SLSSL procedure in Sect. t.2 create augmented model set {8) (w) w))k> with each 6 (w) derived by updating 0 using the weighted version of equation
O' (m( = Vo Σ w. w,Ler (ʃ (In; ,
where denotes the collection of sample weights {wa} within the mini-batch and %h is obtained by corrupting label pair i A j using equation [ estimate 7. we follow Sect. 3.3 and compute the deviation between 8' (w) and 0 by calculating the standard KL divergence Dx ( 6% (w)) using equation 5 Since 8',, (w) is derived by corrupting a label pair * j. this deviation thus corre- sponds to the off-diagona, entry T.j. We further let (w) denote the model derived by updating 0 through a single-step step gradient descent based on the original mini-batch S (without label corruption), and we also compute its deviation from 0 (denoted by D Dxz(0 || (w))) to represent all diagonal entries of f. Since estimate f by taking inverse of the above deviations. for T to be close to iden- tity, ( [e] || 6',, (w)) should be much larger than Dr.. (# (())), Thus, our sample reweighting (RW) loss can be defined in the following contrastive form
LRw DKL (w())
EM DKL (w))
It can be seen that, the denominator in Law describes the performance deviations between the original model and their augmented versions derived by multiple label corruptions. For mini- batch with more clean samples, such label corruptions would lead to larger deviations, compared to the case without label corruption (i.e., the numerator). We thus propose to minimize Law for sample reweighting. In other words, samples leading to higher performance deviations from label corruptions are thus assigned with larger confidences/weights. This serves as our sample reweighting strategy based on SL.SSI..

---

Under review as a conference paper at ICLR 2023
Table 1: Classification accuracy (%) CIFAR-10 across different noisy labeling schemes with
NLL methods based on a single learning model.
Method
No noise
Sym 50%
Sym 70%
Asym 30%
Asym 40%
Bascline (cross-entropy)
92.53+0.07
79.87=0.29
70.93+0.33
86.17=0.17
83.95=0.36
F-correction (Patrini et al. 2017
92.63110.12
81.22.1.04
73.74.10.46
88.28.10.23
87.4510.24
MINT (Li et at. 2019)
93.76+0.03
87.18+0.11
80.76+00.29
91.84++0.28
89.57+0.12
Our SL.SSL (F-step only)
93.86+0.08
89.11+0.08
81.22+0.42
92.67++0.00
91.12+0.10
Our SLSSL (EM)
94.17.10.02
90.1310.16
83.91.10.30
92.14
91.78.10.
3.5 SLSSL AS AN EM-LIKE ALGORITHM
It is worth noting that, we can further integrate the proposed model training (Sect. 3.3 and sample reweighting (Sect. (.44 schemes as an EM-like algorithm for NLL.. To be more specific, the E-step focuses on model training with sample weights being fixed, while the M-steps aims to reweight training samples with the derived model. In practice, we first randomly initialize and train from scratch by optimizing Lsst. in equation and LCE in equation on the unweighted training dataset (w,, = 1 for all n). After obtaining the best model through validation (usually the model 8*), we then fix its parameters and utilize it to optimize all by optimizing equation 9 The updated sample weights are then applied in the model training again. As confirmed by our experiment, this alternative optimization strategy would further boost the NLL performance.
4 EXPERIMENTS
4.1 DATASETS AND SETTINGS
Datasets We conduct experiments on the datasets of CIFAR-10 Krizhevsky & Hinton 2009).
CIFAR-1ON (Wei et al. 2022). , and Clothing (Xiao al. 2015). 2015) The CIFAR- 10 dataset contains 50K training images and 10K test images from 10 categories. We follow (Li et al. 2019: 2020) to corrupt the training labels by two types noise: symmetric and asymmetric. For symmetric noise, each sample has fixed probability to be labeled uniformly into other classes. For asymmetric noise, transition patterns (bird - airplane, cat ++ dog. etc.) are proposed to simulate the class-dependent noise. Fig. a) shows the transition matrix used to construct noisy CIFAR- 10 dataset with 40% asymmetric noise. CIFAR-ION is extended CIFAR-10, 10, with each training image containing three human-annotated labels (denoted as Random-i, E {1, 2, 3)). The three noisy labels for each image are further uggregated by majority voting (denoted as Aggregate) and randomly picking one wrong label if any (denoted Worst). Clothing LM consists of 1M training images from 14 categories of clothes. The labels are extracted from the surrounding texts of images and are thus practically noisy. We use the 14K clean validation set and the 10K test set.
Implementation details For both CIFAR-10 and CIFAR- we follow DivideMix (Li et al. 2020) adopt Pre-Act ResNet-18 network (He et al. 2016) (and its hyperparameters) We also integrate our SLSSL sample reweighting scheme (i.e., the M-step in Sect. 3.4 into the co- dividing step of DivideMix for evaluation. For Clothing IM, we again follow DivideMix and use an mmaerrnt-pretrainee ResNet-50 network. To compare with recent state-of- offthe--rt art methods, we reproduce the experiment results for DivideMix on all the three datasets, as well as Forward Loss Correction (F-correction) (Patrini et al.) (017) and MLNT (Li et al. 2019) on CIFAR-10. Please refer to Appendix A for the more details.
4.2 QUANTITATIVE RESULTS
Comparisons with single-model approaches We first compare the proposed SLSSL, with single- model NLL approaches using the same Pre-Act ResNet-32 backbone, including F-correction (Pa- trini et al. 2017) and MLNT (Li et al.) 2019). The performances are summarized in Table Here, we follow the same procedure in MLNT to generate symmetric and asymmetric noisy labels from CIFAR-10 10, and report the mean and standard error values of test accuracy across runs. As can be seen from the table, with SLSSL-based model training along (i.c., E-step in Sect. 3.3). our frame- work achieved significant performance improvement across all noise ratios. By further applying
7

---

Under review as a conference paper at ICLR 2023
Table 2:
Classification accuracy (%) CIFAR-10 across different noisy labeling schemes with
NLL methods adopting dual models (i.e., co-training based approaches).
Noise type
Sym
Asym
Method/Noise
20%
80%
90%
40%
Mean
Best
96.1
94.6
93.2
76.0
93.4
90.66
DivideMix el 2020
Last
95.7
94.4
92.9
75,4
92.1
90.10
Best
96.3
95.6
93.7
35.3
94,4
83.06
DesSug Nishi al. 2021
Last
96.2
95.4
93.6
10.0
94.1
77.86
Best
93.5
90.3
76.4
57.3
92.3
81.96
SLSSL (E-step only, single-model)
93.1
90.1
76.1
57.0
91.8
81.62
SLSSL (EM single-model)
Best
95.8
94,4
93.2
67.0
93.1
88.70
Last
95.2
94.0
92.8
61.5
92.4
87.18
Best
96.3
95.0
93.4
77.5
94.2
91.28
SLSSL (EM dual-model)
Last
96.2
94.8
93.2
76.7
93.9
90.96
Table 3: Classification accuracy (%) on CIFAR-ION with different human-labeling schemes.
Method/Noise type
Aggregate
Random-1
Random-2
Random-3
Worst
Mcan
DivideMix (Li et al. 2020
Best
95.2
95.5
95.5
95.5
93.0
94.94
Last
95.0
95.1
95.2
95.2
92.6
94.62
Best
95.7
95.5
95.8
95.3
93.3
95.12
Our SLSSL
Last
95.6
95.3
95.5
95.1
93.1
94.92
sample reweighting (i.e., M-step in Sect. 3.4.. the performances were further improved by large margin, which confirms the effectiveness of our proposed SLSSL as sample reweighting strategy.
Comparisons with state-of-the-art dual-model approaches Next, we compare the performance of SLSSL with recent state-of-the-art methods utilizing two networks in the co-training fashion, including DivideMix (Li et al. 2020). and DM-AugDesc (Nishi et al. 2021). Table 2 lists the performance comparisons on the synthetic noisy datasets generated from CIFAR-10. Following DivideMix and DM-AugDesc, we report both the best test accuracy across all training epochs, and the average test accuracy over the last 10 epochs. As can be seen from the table, our frameworks performed favorably against state-of-the-art methods, including DivideMix on which we apply our SLSSL. is worth noting that, DM Aegee also adapted DivideMix and focused on searching for the best augmentation strategies for NLL. On the other hand, we only apply standard augmentation techniques such as random cropping and horizontal flipping in our experiments.
Table 3 lists the performance comparisons between SLSSL and DivideMix (Li et al., 2020) on the CIFAR- 1ON dataset. It can be seen that our method outperformed DivideMix on this challenging human-annouee annotated dataset. Table 4 further presents results the Clothing IM dataset, which con- firms the effectiveness of our proposed SLSSL over state-of-the-ar methods.
4.3 VISUALIZATION AND ANALYSIS
Noise transition matrix for NLL We show the noise transition matrix T estimated based on Sec. 3.3 on the CIFAR-10 dataset with 40% asymmetric noise. We first trained a model by applying our E-step for 100 epochs, and then conducted the M-step for 20 epochs to assign weights for all training samples. Based the sample weights, we then trained our model again using E-step for 100 epochs. As can be seen from Fig. 5()), the noise transition matrix estimated after the first E-step fairly close to the ground-trut truth showed in Fig. 5).). In Fig. Sic),, the other hand, the estimated noise transition matrix becomes much closer to an identity matrix after a complete E-M-E cycle, indicating the effectiveness of our M-step in assigning sample weights.
Reweighted samples Finally, we show the sample weights optimized based our SLSSL the end of three successive M-steps on CIFAR-10 with 40% asymmetric noise. As can be seen from Fig. 6, the optimization progresses, the sample weights of clean and noisy samples become in- creasingly separable (with AUC scores reported in the figure). This further confirms the effectiveness of our SL. for sample reweighting in NLL..
8

---

Under review as a conference paper at ICLR 2023
Table 4: Classification accuracy (%) Clothing M. Note that DM-AugDese (Nishi al. 2021) is designed to search for best augmentation techniques for NLL, while we simply perform random cropping and flipping in SLSSL.
Methed
Test Accuracy
Cross- Entropy
69.21
F-correction Parrini et
2017
MLC. MaCg 2020
71.06
MLNT (Laerar 2019
73.47
T-Reviston eL d.. 2020
74.18
DivideMix
74.76
DM Aug (Nisin al.
2021
75.11
Jo-SRC Yao ct 2021
75.93
DivideMix (reprodueed)
74.34
Our SLSSL
74.51
1400
E30O
CODO
SOCO
E323
3230
4:53
CXQ
RODA
OMO
CMO
9431
0.00
SOGO
AEDE
CBS
SORO
SEDI COCS
COCO
EOOO
CEDE
0363
3230
LAST
LIVE
COOS
BODS
BABS
0412
AUDA
GDO
SUDA
AD2I 6003
0000
5:00
0.00
0003
1:40
0.00
0.03
5:50
3650
EXUL
C:06
04E4
0000
6:00
DROS
0:04
der Co0o
8:00 0,00
6:03
5000
0350
0363
8,02
M022
O8L7
6015
6:12
MGD
OR22
6009
0.03
0.00
0:00
5000
CNTT
0:11
talu- (.00
8000
0,00
6300
(00)
1000
030
1.00
0400
GURL
ADRU
0.06
0300
5000
0.000
0800
1.00
1053
0001
6002
02
dia
GORL 3002
090
Figure 5: (a) The ground-truth transition matrix for constructing the noisy label dataset with 40% asymmetric noise from CIFAR-10; (b) our estimated matrix after the first E-step, which close to the ground truth; (c) our estimated matrix after complete E-M-E cycle (which is close to the identity matrix, implying that our SLSSL assigns proper sample weights for cleaning the training data).
Wy,
Mi the stop
Wy, Mistop
(6)
(C)
Figure 6: Empirical distributions of sample weights derived at the end of three consecutive M-steps (from (a)-(c) for CIFAR-10 with 40% asymmetric noise. Note that AUC scores are shown each sub- fub-figure, quantitatively verifying separation between clean and noisy labeled data.
CONCLUSION
In this paper, we proposed set-level self-supervised learning (SLSSL) to address noisy label learning (NLL) tasks. SLSSLL perffrs self-supervised learning mini-batch levels without prior knowledge on noisy label distribution. By corrupting the labels of each training mini-batch and applying cor- responding consistency constraints, our SL.SSL enforces the model to exhibit sufficient robustness toward labeling noise. Moreover, the proposed SLSSL can also be utilized for sample reweight- ing technique based on a novel high-impact principle. Finally, we demonstrate that the proposed learning scheme can be viewed models. With experiments conducted on synthetic (CIFAR-11) 10) and real-world (CIFAR-101) and Clothing noisy label data, the effectiveness of our proposed SLSSL can be sucessfully verified.
expectation expetation-maxximizzion (EM) algorithm for training NLL

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML., 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look at memorization deep networks. In ICML, 2017.
Benoit Frenay and Michel Verleysen. Classification in the presence of label noise: survey. In TNNLS, 2014.
Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In CVPRW, 2021.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017.
Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NeurIPS, 2018.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In NeurIPS, 2019.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018.
Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In CVPR, 2022.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. In Master's thesis, University of Toronto, 2009.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Learning to learn from noisy labeled data. In CVPR, 2019.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In /CLR, 2020.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. In ICML, 2022.
Eran Malach and Shai Shaley-Shwartz. Decoupling "when to update" from "how update". In NewrlPS, 2017.
Kento Nishi, Yi Ding, Alex Rich, and Tobias Hollerer. Augmentation strategies for learning with noisy labels. In CVPR, 2021.
Diego Ortego, Eric Arazo, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Multi-objective interpolation training for robustness to label noise. In CVPR, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: loss correction approach. In CVPR, 2017.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Gil Lee. Learning from noisy labels with deep neural networks: A survey. In acXiv=2000.008/99v 2021.
10

---

Under review as a conference paper at ICLR 2023
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, 2018.
Haobo Wang, Ruixuan Xiao, Yiwen Dong, Lei Feng, and Junbo Zhao. Promix: Combating label noise via maximizing clean sample utility, In arXiv:2007.000 007.0099900 089966,,,, 2021.
Zhen Wang, Guosheng Hu, and Qinghua Hu. Training noise-robust deep neural networks via meta- learning. In CVPR, 2020.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, 2020.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In ICLR, 2022.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurIPS, 2019.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
Yazhou Yao, Zeren Sun, Chuanyi Zhang, Fumin Shen, Qi Wu, Jian Zhang, and Zhenmin Tang. Jo-src: A contrastive approach for combating noisy labels. In CVPR, 2021.
Yu Yao, Tongliang Liu. Bo Han. Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation eπror for transition matrix in label-noise learning. In NeurlPS, 2020.
Xingrui Yu, Han, Jiangchao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In JCML, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. ICML, 2022.
11

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
Single-model experiment on CIFAR-10 CIFAR-IO, we follow previous works (Tnaka et al., Li et al. 2019) and split the 50K training samples into 45K training and 5K validation sub- sets. We then introduce symmetric or asymmetric label noise to the training subset as described in Sec. 4.1 All models are trained on the noisy training subset, with hyperparameters being chosen based on the model performance on the clean validation subset. After hyperparameter tuning, we then introduce label noises to all 50K training samples and rerun all experiments for comparison.
2018
We develop our SLSSL algorithm based on the implementation of MINT (L et al. 2019), which adopts a Pre-Act ResNet-32 network. We follow most of the hyperparameters from MLNT, and train the model for 300 epochs using SGD with an initial learning rate 0.02 (divided by 2 after every 50 epochs), a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128. The learning rate " used for obtaining the augmented models is set to 0.1 in equation 2 For our SLSSL framework with both E and M steps, we set yj 0.001 in equation 8 and conduct the M-step for 5 iterations (3 M-step epochs per iteration) at every 50 E-step model training epochs: (50, 100, 150, 200, 250).
As for other single-model methods, we simply train the model for 300 epochs using standard cross- entropy classification loss as the Baseline, and add the ground-truth noise transition matrix for F- correction (Patrini et al. 2017). The MLNT results are reproduced by directly using their imple- mentation.
Dual-Model co-training for CIFAR-10 and CIFAR-ION We follow the same principle from the above single-model experiment on CIFAR-IO for hyperparameter tuning, and directly apply the best set of hyperparameters for the 40% asymmetric noisy dataset to CIFAR-ION.
We integrate our SLSSL algorithm into the co-training phase of DivideMix (Li et al. 2020) based their implementation" and also follow most of their hyperparameters. We first train two Pre-Act ResNet- 18 networks separately for 10 epochs in the warm-up phase, and then enter the co-training phase for the remaining 290 epochs. At the 150 epoch, we replace DivideMix's 's GMM-based reweighting procedure on the 2nd network by our SLSSL-based sample reweighting algorithm, and conduct the M-step for 60 iterations (2 M-step epochs per iteration) every 5 model training epochs: {150, 155, 29.). Based the derived sample weights, we then follow standard DivideMix and apply co-dividing, co-refinement, and co-guessing techniques to train the two networks.
Dual-Model co-training for ClothingIM For Clothing M. we also follow DivideMix (Li et al, 2020) and train our model on the 1M (noisy) training dataset, with hyperparameters being chosen based on the 14K (clean) validation subset. Similar to CIFAR-10, we integrate our SLSSL algorithm into the co-training phase of DivideMix. We start our SLSSL-based sample reweighting (i.e., the M-step in Sect. 3.4) at epoch 70. Since only 32K samples are randomly selected per training epoch, conduct M-step for every epoch, and set the total epoch number in the co-training phase as 80. The learning rate is set to 0.002, and decays with a factor of 10 after the epoch 40. All the rest hyperparameterss follow DivideMix.
Algorithm We provide the pseudo codes for our SLSSL in Algorithm I for model training 33. E-step) and Algorithm sample reweighting (3.4) M-step). The two algorithms can be combined as EM-like iterative training strategy described in Sect. 3.5
B VISUALIZATION ON CIFAR-10
We provide additional training statistics to further validate the proposed methods. In Fig. aa), we show the Area Under the Curve (AUC) for clean/noisy sample classification on CIFAR-IO with 40% asymmetric label noise in the single-model experiment. As can be seen, our M-step is able to effectively identify clean/noisy samples across successive M-steps, even for bird samples with 40% samples being mislabeled as airplane, and for cat samples which could be heavily confused with dog samples with 40% samples in each class being mislabeled one another. In Fig. 7 ().
'https//github.com// LJuuman 1992/MLNT hhaps//ggthub.coo// JiJunnn 1 DivdDixid
12

---

Under review as a conference paper at ICLR 2023
Algorithm SLSSL-E (model training in Sect. 3.3)
1: Input: Noisy training dataset = ((nn In, wa)} (fixed {w,, }), learning model EMA model 0* number of E-step epochs P. number of label corruption for each mini-batch M, estimated noise transition matrix at the et epoch Te. estimated noise transition matrix T.
2: if first E-step then
3: Random initialize
4: Initialize EMA model l" = 0
5: Initialize T' identity
6:
Initialize all sample weights Wn =
7: end if
8: for 1 to P do
9:
while not done
10:
Sample mini-batch S = (Lin Dn: w.) ) from D for m = to M do
12:
Randomly sample class pair (i,
13:
Obtain augmented set S {(In Vy Wn wn)]-= by corrupting (i,j) in S (Eq.
14:
Derive augmented model 6', from @ by single-step gradient descent on S' (Eq.
15:
Compute the KL divergence DKL between 6% and (Eq. 4)
end for
17:
Compute set-level self-supervised loss Lsst. (Eq.
Update by minimizing LSSL
Compute classification loss LCE and then correct it using T (Eq.
20:
Update A by minimizing Leg
21:
Aggregate DxL'S derived by corrupting (i.j) to estimate the (i.j) entry of Ta (Eq. 6)
22:
end while
23:
Update T using EMA of To
24: Update 6 using EMA of
25:
end for
Algorithm SLSSL-M (sample reweighting in Sect. 3.4)
1: Input: Noisy training dataset D = {II, In. wx) [.) learning model (fixed), number of M-step epochs Q, number of label corruption for each mini-bateh M.
2: for to do
3: while not done do
4: Sample mini-batch- S
{Zo. Un. w.,) w,)) 1N from D
5: for m = to MM do
6:
Randomly sample class pair (i,j)
7:
Obtain augmented set S" = [(In, V. ) by corrupting (i.j) in S (Eg
8:
Derive 6' (w) from by weighted single-step gradient descent on S" (Eq. 8
9:
Derive (w) from @ by weighted single-step gradient descent on S (Eq. 8
end for
11:
Compute the sample reweighting loss Raw (Eq,
IN 12: Update { by minimizing Law
13:
end while
14:
end for
show the training and testing accuracy values. As can be seen, the accuracy improved after each iteration of M-step epochs 50 and above, further validating the effectiveness of our SLSSL-based sample reweighting procedure.
C ADDITIONAL EXPERIMENT RESULTS
In Tables and results of DivideMix (Li et al. 2020 and DM- AugDesc (Nishi et al. 2021 were directly copied from their papers (i.e., standard deviation were reported by neither works). Here, we further conduct additional runs with different random seeds for our SL.SSI. and Mix selected noisy settings from CIFAR-10 10 and CIFAR- 1-N. The results
CIFAR-10 and CIFAR-1ON
13

---

Under review as a conference paper at ICLR 2023
all
training testing
3.50
bird
50.0
Accuracy
00.0
AUC
3.80
0.70
10
230
300
Accumulated Mistep Epochs
E- E-step Epochs
(4)
(b)
Figure 7: Evaluation on CIFAR-I0 with 40% asymmetric label noise. (a) Area Under the Curve for clean/noisy sample classification based the sample weights [wa] } obtained by our M-steps. The X-axis indicates the accumulated M-step epochs over 5 iterations (3 M-step epochs per iteration) E-step epochs from 50 up to 250. Note that our M-step is shown identify clean/noisy samples across successive M-steps by assigning proper weights, improving both Netl and Net2 in the co- training process. (b) Training and testing accuracy values the first network over all 300 training epochs. The accuracy is improved after each iteration of M-step at epochs 50 and above.
Table 5: Classification accuracy (%) on CIFAR-10 10 across different noisy labeling schemes with NLL methods adopting dual models (i.e., co-training based approaches). We report results over independent runs.
CIFAR-10
CIFAR-ION
Method
Asym 40%
Aggregate
DivideMix (Li al. 2020 (reproduced)
Best
93.20 0.20
95.37+0.12
Lust
93.25=0.15
95.10+0..08
Best
94.20=0.10
95.7310.12
Our SLSSL
Last
93.75-0.15
95.57+0012
are listed in Table 5 Following the setting of DivideMix (Li et al. 2020), we report the best test accuracy (Best) and the averaged test accuracy over the last 10 epochs (Last). From this table, it can be seen that the improvements of our SLSSL over DivideMix were statistically significant.
CIFAR-100 To provide additional performance evaluation, we also add experiments on CIFAR- 100. We select two noise types (20% and 50% symmetric noise) and report results from single run using the same random seed to ensure the same noise setting between DivideMix and our SLSSL. As can be seen from Table our SLSSL again outperformed DivideMix and also MD-DYR-SH et al.
AAazo
2019) on CIFAR- T00.
D ADDITIONAL COMPARISONS
Compare to instance-level SSL approaches As discussed in Sect. it is not clear whether existing instance-level SSL techniques would result in robust representations when tackling the NLL tasks. To make our discussions and comparisons more complete, we compare to MOIT (Ortego et al., 2021). recent instance- based SSL approach to NLL, on CIFAR-10. For fair comparisons,
method
follow MOIT to adapt single PreAct ResNet-18 188 network, and report the test accuracy in the last training epoch. As can be seen fromTable 7 our SLSSL reached comparable performance for asymetric noise 40%, while outperformed MOIT with significant margins for different symmetric noise levels.
Compare to recent related works SOP Liu et al. 2022) proposes to model the label noise and leamn to separate it from the data by enforcing its sparsity, and serves one of the most recent works of NLL. However, as can be seen from Table g our SLSSL still performs favorably against SOP, especially the more challenging Clothing dataset.
14

---

Under review as a conference paper at ICLR 2023
Table 6: Classification accuracy (%) CIFAR-100 across different noisy labeling schemes with NLL methods adopting dual models (i.e., co-training based approaches). For DivideMix (Li et al. 2020 and our SLSSL, we report result from a single run using the same random seed.
Method
Sym 20%
Sym 50%
Best
73.9
66.1
MD-DYR-SH Arazo et al.
2019 (reported)
73.4
65.4
Best
77.50
74.20
DivideMix Li et al.
2020 frepooducdd)
77.00
73.80
Best
78.08
74.34
Our SLSSL.
Last
77.83
73.95
Table 7: Classification accuracy (%) on CIFAR-10 across different noisy labeling schemes from
MOIT (Ortego al. 2021 and our SLSSL.
Sym 20%
Sym 80%
Asym 40%
Mean
MOIT (Ortego al. 2021 (reported)
94.08
75.83
93.27
87.73
Our SLSSL
95.18
92.82
92.23
93.41
E LIMITATIONS
Since our proposed SLSSL can be viewed as a unique meta-learning scheme on existing NLL meth- ods like DivideMix (Li et al. 2020), we expect longer computation time during training. However, take CIFAR-10 for example, DivideMix took 15 hours to train using a single Nvidia TITAN-V GPU, while the full version of our SLSSL (i.e., implemented can seen that, the overall computation time of our SLSSL still in the same order of that of SOTAs like DivideMix.
EM algorithm) required 25 hours. It
Also, as noted in (Patrini et al. 2017 Hendrycks et al. 2018: Xia et al., 2019; Wang et al. 2020 Yao et al. 2020), NLL methods based class-wise noise transition matrix estimation share the limitation that the number of classes for NLL would be reasonable (e.g., 100 in CIFAR- 100). This avoid the potential problem of estimating large noise transition matrix. Sharing the concern of the above works, this would also among the current limitation of our work.
15

---

Under review as a conference paper at ICLR 2023
Table 8: Classification accuracy (%) on CIFAR-ION ION (Aggregate) and Clothing IM from SOP (Liu al. 2022 and our SLSSL.
CIFAR- 10N (Aggregate)
Clothing IM
SOP (Liu et al. 2022 (reported)
95.61
73.5
Our SLSSL
95.73
74.51
16


Reviews:
[{"review_id": "wD3BtVvLys", "reviewer": "Reviewer_rqJk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of learning with noisy-labels. By combining different existing approaches, such as noise transition matrix estimation and sample reweighing, with a new set-level self-supervised learning pipeline, the proposed approach achieve strong performance on NNL (noisy label learning) vision benchmarks. The method can also be seen as an expectation-maximization (EM) algorithm.", "summary_of_the_review": "The proposed method is conceptually interesting and valuable for the community. However it is hard to understand the contribution of all the components of the proposed method and the results are not significantly better compared to the current state-of-the-art. The paper would be greatly improved if the benefit of using a self-supervised component for learning with noisy-labels was clearer.", "strength_and_weaknesses": "\nStrengths:\n\n1) The paper is clearly written and does a great job at explaining the proposed method in details.\n\n2) The proposed method is interesting and introduces the notion of set-level self-supervised learning, and invariance to label corruption, which is a new concept in the field of learning with noisy labels.\n\n\nWeaknesses:\n\n\n1) Calling the method self-supervised feels odd given that the task to solve is (noisily) supervised in itself, and that the gradients from the $\\mathcal{L}_{CE}$ loss are retro-propagated into $\\theta$.\n\n2) How does the model perform without $\\mathcal{L}_{CE}$ ? In particular, it would be interesting to train the model with only with the self-supervised loss and evaluate the model on a linear frozen classification task, similar to what is done in standard SSL [1].\n\n3) How do you balance $\\mathcal{L}_{CE}$ and the self-supervised loss ? More generally, it would be great to have an ablation on the different criterion of the method, in order to better understand the role that each one plays.\n\n4) The experimental results are a bit weak. Comparison with several recent methods cited in the paper are missing. In particular none of the methods mentioned in the \"Self-Supervised Learning for NLL\" paragraph of the related work section are compared in the experiences. For exemple, Jo-SRC [2] achieves 75.93\\% test accuracy on the clothing-1M dataset against 74.51\\% reported by SLSSL in the paper.\n\n5) It would be great to have results on the Cifar-100 dataset which has significantly more classes than Cifar-10 and Clothing-1M.\n\nRemarks and questions:\n\n1) Do you apply different augmentations (Random cropping, color jittering) in the batches S and S' ?.\n\n2) \"most SSL strategies are performed at the instance level, regardless of the correctness of its label\". I don't understand this sentence, as there is no label in SSL.\n\n3) The parallel made with the EM algorithm feels a bit far-fetched, would it be possible to provide a more mathematical comparison ?\n\n[1] T. Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, ICML 2020\n[2] Y. Yao et al., Jo-SRC: A Contrastive Approach for Combating Noisy Labels, CVPR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "v_mRwziFZr1", "reviewer": "Reviewer_szQw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new set-level self-supervised learning (SLSSL) method for combating noisy labels. By corrupting the labels of each training mini-batch, the proposed SLSSL enables the model to exhibit sufficient robustness. The comparison with SOTA methods demonstrates the effectiveness of the proposed method.", "summary_of_the_review": "I do not have major concerns on this paper. It seems that the proposed algorithm works well.", "strength_and_weaknesses": "Pros:\n\n1. The idea of conducting set-level self-supervised learning for tackling label noise is novel and interesting.\n2. The proposed algorithm is rigorously explained.\n3. The experiments are convincing.\n\nCons:\n\n1. I’m not fully understand the sentence “As for instance-level pretext tasks (e.g., rotation prediction or contrastive-based instance discrimination), they are expected to produce proper representations from unlabeled data; it is not clear whether such techniques would result in robust representations when handling input data with noisy labels” in the introduction. In fact, the pretext tasks usually do not depend on labels, so I’m not clear why the data with noisy labels might affect the model output aided by this technique. I cannot see the connections between these two things.\n2. It would be better if pseudo code of the proposed SLSSL can be provided.\n3. I’m not fully understand why the designed set-level augmentation and label corruption way (e.g. Eq. 1) can help resist label noise. It would be more helpful if some explanations can be provided.\n4. In fact, for label noise learning, the estimation of label transition matrix is a quite challenging task. I want to see the estimation accuracy when the data dimension is high. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "tBxV9g1ihT", "reviewer": "Reviewer_xdaD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an SSL method to learn from noisy labeled data.  Instead of the previous method, which is based on instance-level SSL, this method is a set level. Authors proposed to learn M models (with parameters $\\teta^'_i=(1:M))$ ) wherein in each of the models, the labels of two classes are corrupted. Finally, all models, and also the primary model, are forced to have the same behavior concerning the same inputs.  The final is robust aginst the models and explicitly learning the nosie transition matrix.", "summary_of_the_review": "The idea is novel, and the paper is well-written. ", "strength_and_weaknesses": "+The idea is very interesting. It has been shown that the model efficiently works on noisily labeled data. As far as I know, there has been no similar work previously. \n+Results are promising and confirm the feasibility of the idea. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dBQ3hFD67AI", "reviewer": "Reviewer_fPbP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on set-level self-supervised learning (SLSSL) and proposes a new method for this problem with the same name SLSSL. In order to eliminate the bad effects of corrupted labels, SLSSL augments a minibatch by corrupting labels and obtains a augmented model. Then SLSSL enforces the consistency between outputs from the augmented model and the weight-averaged model. The paper then shows how to use SLSSL in model training: the consistency statistics are further translated into noise transition matrix, which refines classification prediction and helps learn the parameters of the model.", "summary_of_the_review": "The idea is interesting, but the paper still has some problems. It is not qualified right now, but has potential to become better after some revision.", "strength_and_weaknesses": "Strength:\n\nThe paper proposes a novel method. The idea of putting SLSSL into a EM-like algorithm is interesting.\n\nWeakness:\n\nThe name of the problem is somewhat confusing. It is suggested to explain the difference between set-level self-supervised learning and supervised contrastive learning.\n\nSLSSL acts as part of the whole learning strategy. It is advised to give a new name to the whole strategy, which may help readers understand the relation between the whole training process and SLSSL better. \n\nWhy should Section 3.2 and Section 3.4 be separated into two parts? Also, the information shown in Fig 2 is all included by Fig 4 and the two figures are very similar. It is suggested to think about the structure of Section 3.\n\nIt is advised to explain why SLSSL is only compared with some weak baselines.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LPwJdiSztd1", "reviewer": "Reviewer_XcmF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studied how to address the negative impact of noisy labels during training. To tackle this challenge, the authors propose a set-level self-supervised learning loss, which augments a set of images  instead of one instance with label corruption and then maximizes the agreement between two augmented sets.  In addition, the authors show that the proposed learning strategy can also be used for estimating the noise transition matrix and sample selection following typical ways of noisy-label learning.  Experiments and ablation studies demonstrate that the proposed method has better performance than several baselines.\n\n", "summary_of_the_review": "The paper investigated an interesting problem with an easy-to-follow method. But the experiments and more analysis should be provided to demonstrate the idea. ", "strength_and_weaknesses": "Strengths:\n1. The paper targets an interesting and relevant problem . \n2. The idea of using set-level self-supervised learning  for noisy-label learning is simple and reasonable. The presentation of the proposed method is clear and readable. \n3. Experimental results show that the proposed schemes offer gains over the baselines. \n\n\nWeakness:\n1.  Baselines are not enough and the contribution is not clear. There are some existing works which studied the same problems[1, 2, 3]. It would be nice if the authors can discuss the relationships between these existing works and the approach in this paper.   Experimental comparisons with some of these works would be desirable. \n2. Experimental results on more datasets and configurations should be included.  Only two datasets are considered for evaluation, which make it hard to estimate the generalisability  and robustness of the proposed approach. The effect of different data augmentation techniques on the proposed approach can also be discussed in the paper or in the  Appendix. \n3. Although the authors provide preliminary visualization of the noise transition matrix and the analysis of  sample weights. I think the paper's quality will improve  if the authors can investigate an in-depth  analysis and theoretical justifications on why the use of set-level self-supervised learning improves noisy-label learning  \n4. The authors do not address the limitations of their work. \n\n\n[1] Robust Training under Label Noise by Over-parameterization, ICML 2022\n[2] ProMix: Combating Label Noise via Maximizing Clean Sample Utility\n[3] Beyond Images:Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Features, ICML 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6Cr1e_FB1FB", "reviewer": "Reviewer_yAaK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the task of noisy label learning (NLL), which proposes set-level self-supervised learning (SLSSL) to model noisy label data.\n\nSLSSL performs self-supervised learning at mini-batch levels with observed noisy labels. \n\nThe proposed method relabel the samples from label i using label j, which is called label corrupt, an augmentation method.\n", "summary_of_the_review": "\nThe contribution of this paper is limited and the paper lacks a reasonable explanation of the proposed modules.\n\nIt is suggested to explain the motivation of this paper in detail.", "strength_and_weaknesses": "\nStrength:\n\nThis paper tries to use set-level self-supervised learning (SLSSL) to address noisy label learning (NLL) tasks.\n\nThe proposed SLSSL can be utilized for sample reweighting technique based on a novel high-impact principle.\n\n\n\nWeakness:\n\nThe paper writing is a bit confusing, for example, “additional supervisory signal can be derived to improve robustness of the model against label noise.” and “it is not clear whether such techniques would result in robust representations when handling input data with noisy labels.” These two points contradict each other.\n\nThis paper lacks sufficient theoretical support. Can you provide a theoretical analysis of this method regarding stability and generalizability?\n\nExperiments should be conducted on more and larger data sets.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3r4ahm5TBl", "reviewer": "Reviewer_HztF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a set-level self-supervised learning method for noisy label learning. The authors propose to enforce model robustness against label corruption by making model predictions (other than the corrupted classes) the same between the original model and augmented models. The noise transition matrix is estimated based on the observation that mislabeled class pairs would lead to fewer differences between the models in the SSL process. The noise matrix is then utilized for sample reweighting. Experiments on two synthetic noisy datasets and one real-world noisy one show the efficacy of the proposed method.\n", "summary_of_the_review": "I think the proposed method is clear and novel. Experimental results are a bit weak. \n", "strength_and_weaknesses": "Strength\n1. The writing is clear and the proposed method seems straightforward and easy to implement.\n2. The idea of combining self-supervised learning for noisy label learning based on simple intuition is novel.\n\nWeaknesses\n1. The improvement over DivideMix seems marginal (Table 2 & 3 & 4). More analysis is needed (statistical test, comparison of training computational cost)\n2. The method is tested with one kind of model (ResNet). Can the method generalize to other architectures like Vision Transformers?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject