S3: Supervised Self-supervised Learning under Label Noise
Authors: Chen Feng
Georgios Tzimiropoulos
Ioannis Patras
Venue: ICLR 2022 Submitted (2022)
Keywords: Learning under label noise
Supervised learning
Self-supervised learning

Abstract: Despite the large progress in supervised learning with Neural Networks, there are significant challenges in obtaining high-quality, large-scale and accurately labeled datasets. In this context, in this paper we address the problem of classification in the presence of noisy labels and more specifically, both close-set and open-set label noise, that is when the true label of a sample may, or may not belong to the set of the given labels. In the heart of our method is a sample selection mechanism that relies on the consistency between the annotated label of a sample and the distribution of the labels in its neighborhood in the feature space, a relabeling mechanism that relies on the confidence of the classifier across subsequent iterations and a training strategy that trains the encoder both with a self-consistency loss and the classifier-encoder with cross-entropy loss on the selected samples alone. Without bells and whistles, such as co-training so as to reduce the self-confirmation bias, our method significantly surpasses previous methods on both CIFAR10/CIFAR100 with artificial noise and real-world noisy datasets such as WebVision and ANIMAL-10N.

Under review as a conference paper at ICLR 2022
S3: SUPERVISED SELF-SUPERVISED LEARNING UN- DER LABEL NOISE
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the large progress in supervised learning with Neural Networks, there are significant challenges in obtaining high high-quality, large-scale and accurately labeled datasets. In this context, in this paper we address the problem of classification in the presence of label noise and more specifically, both close-set and open-set label noise, that is when the true label of sample may, may not belong to the set of the given labels. In the heart of our method is a sample selection mecha- nism that relies the consistency between the annotated label of sample and the distribution of the labels in its neighborhood in the feature space; a relabeling mechanism that relies the confidence of the classifier across subsequent itera- tions; and a training strategy that trains the encoder both with a self-consistency loss and the classifier-encoder with the cross-entropy loss the selected sam- ples alone. Without bells and whistles, such as co-training so as to reduce the self-confirmation bias, and with robustness with respect to settings of its few hyper-parametent our method significantly surpasses previous methods on both CIFAR 100 with artificial noise and real-world noisy datasets such as Vibvis and ANIMAL-ION.
INTRODUCTION
It is now commonly accepted that supervised learning with deep neural networks can provide ex- cellent solutions for a wide range of problems, so long as there is sufficient availability of labeled training data and computational resources. However, these results have been mostly obtained using well-curated datasets which the classes are balanced and the labels are of high quality. In the real- world, it is often costly to obtain high quality labels especially for large-scale datasets. A common approach is to use semi-automatic methods to obtain the labels (e.g. "webly-labeled" images where the images and labels are obtained by web-crawling). While such methods can greatly reduce the time and cost of manual labeling, they also lead to low quality noisy labels.
To deal with noisy labels. earlier approaches tried to improve the robustness of the model using robust loss functions (Ghosh et al. [017]] Zhang & Sabuncu| 2018;) Wang et al.) 2019) or robust regularizations (Srivastava al. 2014 Zhang al.]| Pereyra al. (017), 7).. Goldberger & Ben- Reuven] (2016) tried to model the noise transition matrix between classes while Han et al. (2019); Patrini et al. (201)) Hendrycks et al. (2018) proposed correct the losses of noisy samples. More recently, sample selection methods became perhaps the dominant paradigm for learning with noisy labels. Most of the recent sample selection methods do so, by relying on the predictions of the model classifier, for example on the per-sample loss (Arazo et al. 2019 Li et al. 2020a) or model prediction (Song et al. 2019) Malach & Shalev-Shwartz, 2017). By separating clean samples and noisy samples and subsequently performing supervised training on the clean set, or semi-supervised training on both, sample selection methods achieved the state-of-the-art results in synthetic and real- world noisy datasets.
However, there are three main issues with current sample selection methods. Firstly, the sample selection will be inevitably biased if the models (classifier and feature extractor) are trained with noisy labels - this is immediately apparent in the case that the sample selection based on the loss of the classifier itself (Arazo et al. 2019 Li al. 2020a: Yu et al. 2019 Han et al. 2018). Second, in supervised classification problems, noisy samples usually come from two main categories: closed- set noise where the true labels belong one of the given classes (Set B in Fig. T and open-set noise

---

Under review as a conference paper at ICLR 2022
where the true labels do not belong to the set of labels of the classification problem (Set Cn Fig. Most of the works in the literature, including works that estimate the probabilities of label-exchange between pairs of classes (Goldberger & Ben- Reuven 2016 Patrini al. 2017), that do relabeling based on the model's predictions (Song et al., 2019 Han et al. 2019) or works that adopt semi- supervised approaches (Li et al., 2020a; OOteeoo al. (021) deal with the former and not directly address the latter. However, this is considerable source of noise in real world scenarios, c.g., when training from web-crawled data, where there is less control over the collection of the dataset. Crucially, those works (implicitly or explicitly) relabel all samples and do training based on the new labels of all samples. Those are bound to be wrong for all samples in set C, but also are bound to be wrong for several samples of A and B that are (implicitly of training. For the latter reason, those works do not work well even under heavy close-set noise. Finally, current approaches usually require extensive hyperparameters tuning, often even on per- dataset basis - this is unrealistic in scenarios where there is little knowledge about the types of noise. This is partly due to the complexity of the applied semi-supervised learning method, and partly because of the complicated methods that are employed, such as model pretraining (Zheltonozhskil
explicitly) relabeled in the early stages
jet al.) (222T) and model cotraining (Han et al. 2018 Yu et al. 2019 et al. 2020a), so to deal with self-confirmation bias.
In this paper, we address the problem of training under different types of noise with a simple method-namely Supervised, Self-Supervised learning (S3), with two major components that are clearly separated: a selec- tion/relabelling mechanism that selects/relabels samples so as to construct clean and noisy set (Section 3.3), and a training framework that aims learning a strong feature extractor f and classification head y [Fig. 2 from both noisy and clean samples (Section (.4). In the training stage use off-the-shelf learners and a) train the feature extractor and the classification head using a classical Supervised cross-entropy loss applied only the clean samples - this avoids treating samples identi-
"Tiger"
0
Correcsse
Condlast vobe
Openist nole
Figure 1: Different 'tigers' in an animal
dataset.
fied noisy as if they belonged to one of the given classes as most methods (Arazo al. 2019 Li
eet al. 2020a Ortego et al. |022 Wu et al. (021) implictly or explicitly do; and b) train the feature extractor using Self-Supervised loss, namely the consistency loss between the representations of
augmented version of the sample (as in (Chen & He 2021 applied on all samples this avoids nelse-negatives that are inherent in contrastive-leaming with instance discrimination and different to works that apply the consistency on the label predictions that are unreliable at the first iterations or in the presence of open-set noise. The noisy sample selection mechanism relies on a measure of confidence what we define using the ground truth label of the sample in question and an estimate of distribution of the labels of its neighbours - in order to deal with noisy samples, we adopt a scheme in which the distribution is calculated based both on the ground truth labels and on consistently (over subsequent iterations) confident estimates of the labels. Our method is embedded into a stan- dard MixUp and data augmentation framework, and without bells and whistles, such as co-training of multiple models, it achieves state-of-the-ar results in both synthetic and realistic noise patterns in CI CIFARIO0, ANIMAL-1OON, Clothing I M and Web Vision datasets.
RELATED WORK
Some works focused on sample selection to filter out noisy samples. Jiang et al. (2018) introduced pretrained mentor network to guide the selection of a student network. [Song et al. (2019) evaluates the per-sample losses and identify as clean the top r% of the samples- - the precise ratio 3% depends is either predefined, or is an estimate of the noise level in the specific dataset. Arazo et al. (2019) proposed to model per-sample losses with a Beta Mixture Model (BMM) and split the dataset according to which of the components of the mixture each sample belongs. In a very similar approach, Li et al. (2020a) extended uponArazzo AArao et al. (2019) by introducing semi-supervised learning to fully utilize the dataset. Related to our work, Bahri et al. (2020): Ortego et al. (202I): Wang et al.) (2018) also utilized the feature space for sample selection. Bahri al. (2020) applied KNN for sample selection for closed-set noisy dataset while(Ortego al.
Leaning with noisy data by sample selection
2

---

Under review as a conference paper at ICLR 2022
Stage 1: Sample Selection & Relabelling
3.3 Sample relabelling
Sec 3.3
X,
X X aug_eval .->
Encoderr f
Classifier
Sample selection
Stage 2: Supervised Self-supervised Training
X
Sec 3.4 Supervised
aug train
Encoder
Classifier y
mixup training....
X...
aug strong
Encoder
Projectorp
Predictor
Sec 3.1 Self supervised consistency training...
X.... wuakw
Encoder
Projector
stop grad
Figure 2: S3 consists of two iterative stages: sample selection&relaaelling and supervised self-
supervised training.
(2221) further proposed to relabel samples based on the KNN voting, Wang et al. (2018) proposed reweight samples based its probability of being outliers in open-set noisy dataset.
Self-supervised learning Self-supervised methods attempt to learn good representation without human annotations. In the recent years, the dominant method is contrastive learning with instance discrimination task. MoCo (He al., 2020) an important baseline for current contrastive learning methods, which reuses the memory bank since samples in single mini-bate batch may lead to insuffi- cient negative pairs, and proposes momentum encoder to update the memory bank in real-time to avoid outdated data representation. SimCLR (Chen et al.) (2200 is another important baseline which found that setting mini-batch size to large enough can eliminate the need for memory bank, More recently, SimSiam (Chen & He 2021) and BYOL. (Grill et al., 2020) proposed non-contrastive learning framework which enforce the perturbation consitency between different views, and avoid mode collapse by applying stop-gradient and extra predictor between two representation vectors.
METHOD
PROBLEM FORMULATION
Let us denote with X [ 04 Rd, training with the corresponding one-hot vector labels Y {31 J.V. y; E {0,1)K where K is the number of classes and N is the number of samples. For convenience, let us also denote the index where the one-hot vector y: is one as the label I; (0...,,y) K). Finally, let denote the true labels with Y (v/) Clearly, for open-set noisy label it is the case that y! + y: va 1)K, while for closed-set noisy samples y * ys v E 13
3.2 OVERVIEW OF PROPOSED METHOD
Aiming deal with potential concurrence of both open-set and close-set noise, we view the clas- sification network as an encoder f that extracts a feature representation and classification head g that deals with the classification problem in question. The proposed method, named Supervised (SF--Supervise((33)) learning, attempts to decouple their training so as to deal with possible noise in the labels, by adopting two stage, iterative scheme, outlined in Fig.
In the first stage, we utilize novel sample selection and a novel relabeling mechanism (top block in Fig. 2) that prepares the set based which the classifier should be trained in Stage 2. The selec- tion mechanism is based on the assumption of smoothness labels the feature space, and more specifically, consistency measure that defined based the annotated label of the sample
3

---

Under review as a conference paper at ICLR 2022
in question and the distribution of the labels in its neighborhood in the feature space. Relabeling is performed on samples for which the classifier gives confident predictions consistently across subse- quent iterations. Clearly, the mechanism relies on the quality of the features extracted by the encoder f and should reject samples whose true labels are not in the class set (open-set noise). This stage is explained in Section 3.3 In the second stage (bottom block in Fig. (2). training is performed with two objectives/loses. First, a cross entropy loss on the output of the classifier (i.c., on (f()))) on the samples selected in Stage 1, that updates both the encoder f and the classifier head g. Second, a self-supervision loss that enforces consistency between the representations of different augmenta- tions of the same sample, and which utilizes all samples, that is both noisy and clean- this updates the encoder f and helps learning strong feature space on which the selection mechanism of Stage can rely. By contrast to other methods that, either by using a noise transition matrix or their semi- supervised scheme, implicitly relabel all samples (e.g. DivideMix (Li et al. (020a), MOIT (Ortego cet al., 202I) and use the new labels to learn, in our method the labels in the noisy set are not used at all. This stage is explained in Section 3.4
3.3
SAMPLE SELECTION & RELABELLING
The sample selection and relabeling mechanism are designed so as to construct a clean subset that has as many correctly labelled samples as possible.
Clean sample selection by balanced neighboring voting Let us denote the similarity between the representations f. and f, of any two samples æ; and 2; by it,,2,j = I...,,N, N. In our imple- If و Let us also denote by N₁ the index set of the k nearest neighbors of sample æ; in X based on the calculated similarity. Then, for each sample T, we calculate the balanced label distribution P; E RK in its neighborhood in the feature space, as the normalized sum of its neighbors' labels. Note, that we used at each epoch 4, we use the labels y/ (Eq. 2 that a relabeling mechanism provides. More specifically,
mentation we used the cosine similarity, that is, $iz - S.
p: = π Pp.
(1)
where p/- = 2 ya and π Σn ye With a slight abuse of notation, we denote by л the vector whose entries are the inverses of the entries of the vector π of the class probabilities in the whole dataset- in this way we compensate for global class imbalances. Once the balanced distribution p3 of the labels in the neighborhood of x; is estimated, we define consistency measure c as
((())
(2)
max (:)()
that is the ratio of the value of the distribution p. at the label 1, divided by the value of its highest peak max; (I). Roughly speaking, a high consistency measure o; a sample means that its neighbors agree with the given annotation 4; - this indicates that (; is likely to be correct. By setting a threshold 0, to Cs a clean subset (Xe, Ye) can be extracted from the noisy dataset.
Noisy sample relabelling by classifier thresholding The balanced distribution P. of the labels of the neighbors of a sample constructed by Eq. to some degree affected by the noisy labels of its neighbors. One option is to the computed normalized nearest neighbor distribution p; relabel the noisy samples- for example, relabel the sample x; as arg max p. (Ortego et al., (021). However, this process will introduce self-confirmation bias as we relabel and select samples by relying in both cases on the feature space.
In this paper, we propose to decouple sample selection and sample relabeling. More specifically, given that the classifier is trained in subsequent iterations with relatively clean samples (selected by the mechanism described above), we propose to used its predictions identify samples for which it has high confidence in its prediction, and relabel them if the confident prediction does not agree with the annotated label. More specifically, let us denote by 2 a (x:;) the prediction at epoch t for sample x; By keeping track of the most recent L predictions, we calculate
1 L
Σ
q =
(3)
r'=t-L
4

---

Under review as a conference paper at ICLR 2022
where q; is the average prediction for the sample z4- We then modify all the labels 4, = I,.,,N by thresholding q, at f-th epoch as:
C (max (1) > 0+) arg max (1) + (1 [Imax (1()) > 0.)) *
(4)
we denote the one-hot label corresponding to 1 as y this will be used in Eq. [ By setting high 0,, highly confident sample æ; will be relabeled this can in turn further enhance the quality of sample selection. Note, that we avoid mis-relabelling open-set noise samples as those tend not have highly confident average predictions.
Please note, that similarly to Section 3.1
3.4 SUPERVISED SELF-SUPERVISED TRAINING
For training, we have both supervised learning loss on the clean subset (cross entropy) and self- supervised learning loss (consistency). The latter makes no assumption at all (implicit or explicit) about the labels and therefore can be applied on all samples.
Supervised mixup training of the cncoder-classífer using the clean subset With two random samples #1, y/1 and Ø2 y/2 in the clean subset (Xer e, Ye), a mixed new sample am Um will be gen- erated by Mixup method (Zhang et al. 2017) as:
~ Beta(a, a a), X' = max(A, A), am Xxx + X)22 ym = X + (1 N)y22 (5)
We then apply the normal cross-entropy loss for the new virtual mixed sample:
K yo logz
Loe
(6)
where Zm Softmarro((fa we expect that virtual samples generated by Mixup are further away from the dataset samples thus can alleviate the noise memorization effect (Zhang et al. 2017). This loss is prop-propagated so as to update both the encoder f and the classification head g.
Instead of direct training with samples from the clean subset,
Self-supervised consistency training of the encoder using all samples To fully utilize all the samples, we applied self-consistency loss motivated by recent non-contrastive self-supervised learning methods (Chen & He, 2021] Grill et al. 2020). With a projector head p and prediction head q, we minimize the negative cosine similarity between two different augmented views from the same sample z₁. Denoting the two output vectors as q. & q(p(f(xx)) and p, & p(f(x))))
q Ps
Lue
q 2 Pill2
where æ;1, z2 denotes two different augmented views. Lse bears similarity to the commonly used consistency regularization in semi-supervised learning methods, however, the consistency is en- forced between the projected features rather than the predictions. This allows us to utilize open-set noise also for training whose true labels are not in the label set. Also, we applied gradient stopping and extra predictor as to avoid mode collapse. This loss is back-propagated encoder ff and contributes to learning a strong feature space from both clean and noisy samples. We also investigated on the use of the I.2 loss as distance metric details can be found in Appendix D
to update the
Data augmentations Strong augmentations, such as Cubuk et al. (2020) 2019), have shown to be effective in both supervised and semi-supervised learning (Berthelot et al. 2019 Sohn et al., 2020) and recently, Nishi et al. (202T) validated the benefits of strong augmentations within the DivideMix (Li et al., (2020a) framework. In this work, we define and use three types of augmenta- the original image itself (augmentation type 'none") is used for testing, random cropping and horizontal flipping (augmentation type 'weak'), and the augmentation policy proposed in Cubuk et al. (2019) (augmentation type 'strong'). In the model training phase, by default, we apply 'strong' augmentation for E;1, 'weak' augmentation for x; in Lae, and 'strong' augmentation for I1, I2 in to introduce more variance and alleviate the accumulation of error this is in contrast to most works that do not apply augmentations at similar stages.
tions:
Lee. In the sample selection and relabelling phase, apply 'weak' augmentation
5

---

Under review as a conference paper at ICLR 2022
Balanced sampler In order to address possible class imbalances in the noise-agnostic dataset, we balanced the local distribution of class labels, with the inverse of the global class probabilities vector during sample selection (Eq. [p. Similarly, the extracted clean subset (X.,, Y.) might also potentially suffer from class imbalances. To deal with this, we also use a balanced sampler for training with Lee by oversampling the minority class.
The overall training objective is to minimize a weighted sum of Loe and Lec-
Lee wLic
while for all experiments, we fix 1.
4 EXPERIMENTS
4.1 OVERVIEW
In this section, conduct extensive experiments standard benchmarks with artificial label noise, CIFAR-10 and CIFAR-100, and two real-world datasets, Web Vision and ANIMAL-ION (see Appendix A for details). In Section 4.2.1 we conduct extensive ablation experiments show the ro- bustness of our method w.r.t its hyperparameters with different noise types, noise ratios and dataset. In Section 4.2.2 we conduct extensive ablation studies validate the benefits different modules in our method. In Section 4.2.2 we compared S3 with related works in each single stage. In Section and [4.4 we compared with the state-of-te of-the--ttrrrrr- in synthetic noisy datasets and real-world noisy datasets. Implementation details can be found in Appendix B
4.2 ABLATIONS STUDY
In this section, we conduct extensive ablation experiments to show the robustness w.r.t the few hyperparameters with different noise types, noise ratios and dataset. In addition, to clearly show the contributions of different components and choices in S3 we perform experiments where we replace our training sample selection/rclabelingg components with those from other methods in the literature.
4.2.1 ANALYSING SAMPLE SELECTION AND RELABELLING
We aim at building a framework which is robust noise-agnostic dataset scenario and with minimal number of hyperparameters. In this section, we conduct extensive ablation experiments to show the robustness of the few hyperparameters with different noise types, noise ratios and dataset.
50% sym
90% sym
97.0
96.0
95
95.5
50
80
96.0
90
70
95.5
94.5
85
95.0
94.0
1e150
96.3
L-10
93.0
0.8
0.3
1.0
0.7
0.8
0.9
1.0
0.0
0.8 1.0
8.
8.
(8)
(b)
(c)
Figure 3: Classification accuracy with different 0,, L, l and 0g of synthetic CIFAR10 datasets. (a)
0, [0.7, 0.8, 0.9, 1), 10]: (b) & = [1, 10, 50, 100, 150, 200, 250, 300): (c) 0. = 0., 1.0).
0, and L. in sample relabelling The choice of f, and L controls the sample relabelling quality and proportion. Roughly speaking, the lower the 0,, and L, the more samples will be relabeled in the training process, which also means that possibly more errors will be introduced. In Fig. 3 we reported performance with different combinations of 6 and on the synthetic CIFAR noisy dataset. Genrally, our method achieved superior performance than the state-of-the- -ftth--rrt with different 8, and For example in CIFAR10 dataset with 90% sym noise, the lowest accuracy is 87.789..777 7.9% this surpasses the state-of-the-art by 77%

---

Under review as a conference paper at ICLR 2022
Robustness w.r.t & and 9, in sample selection The number & controls the neighborhood size in the sample selection phase. In Fig. b), we report results with different & for the CIFARIO dataset with 40% asym noise. Except for too small & which is more sensitive the noisy samples. the performance is stable and consistently higher than the state-of-the-art. 0g controls the number of selected samples, with 8g 0 corresponding
sample selection (i.e., all samples considered as clean subset). In Fig. 3 c), we report results with 0,- = [0, 0.8, 1.0]. Removing sample selection leads to severe degradation especially in high noise ratio (90% symmetrie noise), while relatively high 8g giving consistently high performance. We fixed 0, = for experiments.
Peformance of sample selection and relabelling At each iteration after relabelling and before sample selection, S3 creates labeled set which one can find correctly labelled samples (let's denote this subset by_4yy_/ A4), wrongly labeled samples that could be labeled with one of the given labels (subset B¹) and the out of set noise (subset C²). Note, the correspondence with the sets A, and C in the original labeled set depicted in Fig After the sample selection, some will be selected and some not. Ideally, the relabelling mechanism will relabel all samples correctly (i.e., A' = A + B) and, at every iteration, the selection mechanism will select only the correctly laheled samples (i.e., those in A'). In order to show the performance of the selection mechanism, we report in Fig. Aa the F-score for the selection. In order to show the performance of the relabeling mechanism, we show in Fig. bb) the ratio of the correctly labeled samples (i.e., A /(A B + C.. It is clear that the relabelling mechanism succeeds in increasing the size of correctly labeled samples (A') even in the case of heavy or out-of-set noise. It is also clear that in all cases the sample selection mechanism achieves high scores, even the challenging cases of heavy noise.
1.0
2.9
0.
0.7
0.7
20pmm
0.3
505m
8demm
04-
50amm
dcacym
S0ap:0apen
50
300
150
200
250
300
50
100
35.
200
250
300
spoths
spochs
(a)
Figure 4: (a) F-score of the sample selection mechanism; (b) Ratio of samples with correct labels after sample relabelling.
4.2.2 TRAINING MODULES ABLATIONS
Effeets of self-consistency and Mixup In Table we report the effect of self-consistency regular- ization and Mixup in S3 training stage. Removing Mixup decreases the performance, especially in high noise ratio and removing celf-consistency also lead to degradation. Both help prevent memo- rization of wrong selections and to explore all samples so as to improve the model robustness.
Dataset
50% sym
90% sym
40% asym
S3
96.25
94,92
95.97
Method
Clothing IM
40% asym CIFARIO
w/o Mixup
94.13
83.11
93.99
74.91
95.97
w/o self-connistency
95.80
93.31
95.54
wi balancing
74.12
95.39
Table 1: Classification accuracy w/o Mixup and Self-consistency.
Table 2: Bfffec of balancing strategies.
Effect of balancing strategies To alleviate possible dataset class imbalance, we propose two bal- ancing strategies in sample selection and model training phase, respectively. In Table 2 we in- vestigate its effect with different controlled noise in CIFARIO and also a well-known real-world imbalanced noisy dataset Clothing

---

Under review as a conference paper at ICLR 2022
4.2.3 ABLATIONS BY REPLACING COMPONENTS
To demonstrate the contributions of our choices, we substitute components of our method, with components of two state of the art methods, i.e., MOIT and DivideMix, and more specifically, their sample selection and training stages. The results are summarised Table [ is clear that the semi-supervised training scheme that MOIT adopts, in which all samples are (implicitly) relabelled, and the DivideMix sample selection that is based on the classifier's 's prediction, perform significantly worse than ours, especially in the case of high noise ratios.
Sample selection
Training
50% sym
90%ssym
40% asym
60% 5S0( open-set)
$3
S3
96.30
95.20
96.0
94.81
S3
MOIT
95.27
69.52
94.75
94.33
DivideMix DivideMix
S3
95.61
46,52
95.04
94.64
MOIT
95.75
10
91.67
80.41
Table 3: Comparion with other works
4.3
SYNTHETIC NOISY DATASETS EVALUATION
In this section, we compared our method to most recent state-of-the-an methods: DivideMix LLi
ct al.
2020a), LossModelling (Arazo et al., 2019), Coteaching+ (Yu et al. (019), Mixup (Zhagg
et al.
2017). F-correction (Patrini et al., 2017), SELFIE (Song al. 2019). PLC (Zhang et al.
2021
PENCIL Yi Wu 2019), ELR (Liu et al. 2020a), NCT (Chen et al. (202]). MOIT+ OOregg
al.
2021). NGC (Wu a 2021 D. RRL (Li al. (2020b), FaMUS (Xu et al. [2021) GJS (Ghosh
& Lan
(202T). PDLC (Liu et al. 2020b). We show, that the proposed method achieves consistent
improvements in all datasets and all noise types and ratios.
Dataset
CIFARIO
CIFAR100
Noise type
Symmetric
Assymetric
Symmetric
Noise ratio
20%
50%
80%
90%
40%
20%
50%
80%
90%
Cross-Entropy
86.8
79.4
62.9
42.7
85.0
62.0
46.7
19.9
10.1
Co-teaching+
85.7
67.4
655.6
27.9
F-correction
86.8
79.8
63.3
42.9
87.2
61.5
46.6
19.9
10.2
Mixup
95.6
87.1
71.6
52.2
67.8
57.3
30.8
14.6
PENCIL
92.4
89.1
77.5
58.9
88.5
69.4
57.5
31.1
15.3
LossModelling
94.0
92.0
86.8
69.1
87.4
73.9
66.1
48.2
24.3
DivideMix
96.1
94.6
93.2
76.0
93.4
77.3
74.6
60.2
31.5
95.8
94.8
93.3
78.7
93.0
77.6
73.6
60.8
33.4
AugDesc
96.3
95.4
93.8
94.6
79.5
77.2
66.4
41.2
C2D
96.4
95.3
94.4
93.6
93.5
78.7
76.4
67.8
58.7
RRL
95.8
94.3
92.4
75.0
91.9
79.1
74.8
57.7
29.3
NGC
91.6
79.3
62.7
Ours(S3)
96.6
96.3
95.7
94.9
79.7
77.2
70.4
56.8
Table 4: Evaluation of S3 with fixed hyperparameters on CIFAR-10 and CIFAR-100 with closed-set
noise. Results of other models are copied from (Li et al. 2020al Wu et al. 202I).
Evaluation with controlled closed-set noise In this section we compare S3 to the most competi- tive recent works. fable@[howw results on CIFARI0 and CIFAR 100 we note for S3 this is without the use of co- co-training such as in C2D (Zheltonozhskii et al., (2221) and without per-dataset finetuning of the hyperparameters as done in several other methods. Here we report the best results of other methods and the results of S3 with fixed hyperparameters. It is clear that our methods outperforms them, not only in the case of symmetric noise, but also that works very well also at the more realistic asymmetric synthetic noise.
Evaluation with combined open-set noise and closed-set noise Table shows the performance of our method in a more complex combined noise scenario. The closed-set noise are generated symmetric noise while the open-set noise are random samples from CIFAR100. Noise ratio denotes the total noise ratio while the open ratio denotes the proportion of open-set noise. Previous methods that specially designed for open-set noise degrade rapidly when the open-set noise ratio
8

---

Under review as a conference paper at ICLR 2022
Method
Noise ratio
0.3
0.6
Open ratio
0.5
0.5
Methed
WebVision
ILSVRC2012
ILON
Best
87.4
90.4
80.5
83.4
Top1
TopS
Topl
Top5
Last
80.0
55.2
78.0
DivideMix
77.32
91.64
75.20
90.84
ELR
76.26
91.26
68.71
87.84
RoG
Best
89.8
91.4
84.1
88.2
InceptionResNee22
ELR+
77.78
91.68
70.29
89.76
Last
85.9
89.8
66.3
82.1
NGC
74.44
FaMUS
79.40
92.80
77.00
92.76
DivideMix
Best
91.5
89.3
91.8
89.0
RRL
76.3
91.5
73.3
91.2
Last
90.9
88.7
91.5
88.7
ResNer50
EDM
Best
94.5
92.9
93.4
90.6
DivideMix
76.08
ResNet18
ELR
73.00
MOIT+
78.76
Ours(S3)
Best
96.34
96.05
94.97
94.01
Ours(S3)
80.12
92.80
74.84
91.26
Last
96.13
95.95
94.81
93.54
Table 6: Testing accuracy Webvision. Results of other methods are copied from MOIT (Ortego al.
Table 5: Evaluation CIFARIO with com- bined noise. Results of other methods are copied from EDM (Sachdeva et al. 2021.
2021) and NGC (Wu et al. 2021
decreased from to 0.5 (ee et al. 2019 Wang et al. 2018). The performance of method without considering open-set noise like DivideMix (Li et al. (020a) will decrease when the open-set noise ratio is increased. EDM (Sachdeva al. 2021) modified the method of DivideMix to deal with combined noise, however report results that are considerably lower than ours.
4.4 REAL-WORLD NOISY DATASETS EVALUATION
Finally, in Table G Table and Table we show results on the WebVision, Clothing IM and ANIMAL-ION datasets, respectively. To summarize, our method achieves better or comparable performance in relation to the current state-of-the-art in both large-scale web-crawled dataset and small-scale human annotated noisy dataset.
F-correction
ELR
C2D
FaMUS
RRL
DivideMix
ELR+*
AugDese*
Ours(S3)
69.21
69.84
72.87
74.30
74.40
74.84
74.76
74.81
75.11
74.91
Table 7: Testing accuracy on Clothing M. Results of other methods are from RRL Li et al. 2020b) and AugDesc (Nishi et al. 202T). Please note methods with * utilized model cotraining or ensem- bling.
Cross-Entropy
SELFIE
PLC
NCT
Ours(S3)
79.4 + 0.1
81.8 + 0.1
83.4 + 0.4
84.1 + 0.1
88.5 0.1
Table 8: Testing accuracy on ANIMAI-1ON, Results of other methods are from NCT (Chen al., 2021
5 CONCLUSIONS
In this paper we propose method for learning with noisy labels, that relies on sample selection mechanism, relabeling mechanism and a training strategy with multi-objective losses that enable us learn robust features from both noisy and clean samples, and classifier from only clean or robustly relabeled ones. The proposed method a simple framework, does not utilize complicated mechanisms such co-training to deal with self-confirmation bias, and is shown with extensive experiments and ablation studies to robust to the values of its few ppper-paraetter, and to consistently and by large surpass the state-of-the in both open-set and close-set noise.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International Conference on Machine Learning, pp- 312-321. PMLR, 2019.
Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Confer- ence on Machine Learning, pp. 540-550, PMLR, 2020.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. arXiv preprint 191 1.09755, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Yingyi Chen, Xi Shen, Shell Xu Hu, and Johan AK Suykens. Boosting co-teaching with compres- sion regularization for label noise. arXiv preprint arXiv:2104.13766 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp- 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2703-2708, 2021.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. 2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. arXiv preprint arXiv: 180400882..0. 2018.
Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-lcarning from noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5138-5147, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738. 2020.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks labels corrupted by severe noise. arXiv preprint 1802::1802.05300 2018.
10

---

Under review as a conference paper at ICLR 2022
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference Machine Learning, pp.
PMLR, 2018.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning, pp. 3763-3772. PMLR, 2019.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning, arXiv preprint arXiv:2002.07394, 2020a.
Junnan Li, Caiming Xiong, and Steven Hoi. Learning from noisy data with robust representation learning. 2020b.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: (oxivv:778..08800 02862, 2017.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. arXiv preprint arXiv:2007.0015/ 20200,
Yun-Peng Liu, Ning Xu, Yu Zhang, and Xin Geng. Label distribution for learning with noisy labels. In ISCAI, pp. 2568-2574, 2020b.
Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". arXiv preprint arXiv: 1 00010.211,..... 2017.
Kento Nishi, Yi Ding, Alex Rich, and Tobias Hollerer. Augmentation strategies for learning with noisy labels. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 8022-8031, 2021.
Diego Ortego, Eric Arazo, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Multi-objective interpolation training for robustness to label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6606-6615, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Gabriel Pereyra, George Tucker, Jan Chorowski, Fukasz. Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 1ee:::0010000000 2017.
Ragav Sachdeva, Filipe Cordeiro, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. Evi- dentialmix: Learning with combined open-set and closed-set noisy labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3607-3615, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying superviee learning with consistency and confidence. arXiv preprint arXiv:2001.07685, 2020.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. International Conference on Machine Learning, pp. 5907-5915. PMIRR, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):192-11953 2014.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. (688-8699 2018

---

Under review as a conference paper at ICLR 2022
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 322-330, 2019.
Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Nge: A unified framework for learning with open-world noisy data. arXiv preprint arXiv:2108. 11035, 2021.
Youjiang Xu, Linchao Zhu, Lu Jiang, and Yi Yang. Faster meta update strategy for noise-robust deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 144-153, 2021.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 7017-7025, 2019.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International Conference on Machine Learning, pp. 7164-7173. PMLR, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv: 0.4400 2017.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami. and Chao Chen. Learning with feature-dependent label noise: A progressive approach. arXiv preprint arXiv:2103.07756 2021.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv: 0088.00000776 2018
Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Con- trast to divide: Self-supervised pre-training for learning with noisy labels. arXiv preprint arXiv:2103. 2021.
A DATASET DETAILS
CIFARIO 10 and CIFARION 100 both consist of 50K images. Following the standard practice, for CIFAR- 10 and CIFAR- 100, we experimented with two types of artificial noise: symmetric noise by randomly replacing labels of all samples using a uniform distribution; and asymmetric noise by randomly exchanging labels of visually similar categories, such as Horse + Deer and Dog ++ Cat.
large-scale dataset of 1000 classes of images crawled from the Web. Following previous work (Jiang et | 2018 Li et al. [220a]] Ortego et al. [202)), we compare baseline methods on the top 50 classes from Google images Subset of Vision. The noise ratio that is estimated to be around 20%. ANIMAL-ION (Song al.) 2019) is smaller and recently proposed real-world dataset consists of 10 classes of animals, that are manually labeled with an error rate that is estimated to be approximately 8%. ANIMAL- 10N has similar size characteristics to the CIFAR datasets, with 50000 train images and 10000 test images.
WebVision (Li et al., 2017)
B EXPERIMENT DETAILS
We used PresActResNet-18 18 (He et al. 2016) as the backbone for all CIFARI0/100 experiments following previous works. Unlike previous methods that use specific warmup settings for CI- FARTO/CIFARION 100, we train the model from scratch with linear raising 0, from to in 20 epochs. 0, is fixed as 0.8 and the prediction track length L is set to 10 for all CIFAR experiments except in the corresponding ablation part. We train all modules with the same SGD optimizer for 300 epochs with momentum of 0.9 and weight decay of 5e-4. The initial learning rate is 0.02 and is trolled by cosine annealing scheduler by Pytorch. The batchsize is fixed as 128. We set a = 4 for all noise settings in Mixup training.
For Web Vision, we used a standard ResNet18 following MOIT (Ortego et al. 2021) due to the hardware limitation. We train the network with SGD optimizer for 150 epochs with momentum
12

---

Under review as a conference paper at ICLR 2022
of 0.9 and weight decay of le-4. The initial learning rate is 0.02 and is controlled by cosine annealing scheduler. The batchsize is fixed as 64. For Clothing IM, we used ResNet50 following DivideMix (Li et al., 2020a) with ImageNet pretrained weights. We train the network with SGD optimizer for 80 epochs with momentum of 0.9 and weight decay of 1e-3. The initial learning rate is 0.02 and and is reduced to 0.002 after 40 epochs. For ANIMAL-ION, & Zisserman 2014) with batch-normalization following (Song et al. 2019). We train the network with SGD optimizer for 100 epochs with momentum of 0.9 and weight decay of 5e-4. The initial learning rate is 0.02 and and is also controlled by cosine annealing scheduler. The batchsize is fixed as 128. For all datasets, we train the model from scratch with 6₂ 1, while 6, fixed as 0.9 and the prediction track length L is set to 1. We set = I for Mixup. We report averages of at least two runs on a single Nvidia RTX 3090 GPU card.
used VGG-19 (Simonyan
C EFFECT OF AUGMENTATIONS
In Table we show the effect of different types of augmentations the different stages of our method. Our results are consistent with the findings of AugDese (Nishi et al. [22T] that one should use weaker augmentations for sample selection and relabeling, and strong augmentations for train- ing. We can also that the higher the noise ratio, the more prominent the differences are with different augmentation strategies.
Dataset
50% CIFARIO
90% CIFARIO
SSR
N
W
N
W
S
SST
W
S
W
S
W
W
8
S
W
ACC (5)
96. 18
96.45
96.41
96.48
96.14
96.22
93.46
95.13
93.70
94,92
90.92
93.85
AugDese Nishi al. 2021]
95.6
91.9
Table 9: Classification accuracy for different augmentations. N denote 'none' augmentation, W denote 'weak' augmentation, S denote 'strong' augmentation; SSR denote sample selec- tion&relabellingg SST denote supervised self-supervised training.
D DIFFERENT DISTANCE METRICS IN SELF-CONSISTENCY
In S3, we applied negative cosine similarity as distance metric for self-consistency head. Here we also experimented with L2 distance. The results shown in Table I0, where it can be seen that there are small differences, but that the cosine similarity in general better.
Training
50% sym
90%sym
40% asym
60% all(50% open-set)
(negaativ cosine similarity) S3(L2 distance)
96.30
95.20
96.0
94.81
96.04
94.7
96.13
94.26
Table 10: S3 with different distance metric for self-consee self-consistency loss
13


Reviews:
[{"review_id": "n6Pdp2Rrpgz", "reviewer": "Reviewer_bqh4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n- State-of-the art results in different settings.\n- The methods works both with open- and closed-set noise.\n- While the ideas of neighborhood-based sample selection, sample relabeling, and consistency regularization are well-known, the particular implementation is novel and interesting.\n**Weaknesses**\n- The presentation should be improved. The authors explain well-known concepts like softmax gradient but some details of the proposed methods are explained briefly or not at all.\n- There is a number of existing methods that you don't compare to: C2D (Zheltonozhskii et al., 2021), which you cite but do not include in comparison and which outperforms you at least in some setups (e.g., CIFAR-100 90% noise -- 58.45±0.30% as opposed to 56.8% you report); FaMUS (Xu et al., 2021). Similarly AugDesc (Nishi et al., 2021) is mention in Tab. 1 but not in main comparison.\n**Comments and questions**\n- Most of existing papers report top-1 and top-5 accuracy for WebVision training, both on ImageNet and WebVision validation sets. Can you provide those numbers?\n- Comparison with other self-consistency approaches could be helpful (e.g., L2 loss is often used in semi-supervised learning or contrastive loss)\n- Different parameters are compared on different datasets, which makes the comparison hard. You should use same settings (or multiple settings, but for every experiment). Also, you show dependence on $\\theta_r$ but not $\\theta_s$. Why?\n- Using average of at least two runs is better than nothing, but in some cases differences are clearly not significant, e.g. Tab. 1 and Tab. 3. Showing standard deviation would make clear which improvements are significant.\n- Probably balancing would be more effective in unbalanced settings, either synthetic or real-life (maybe Clothing1M, Xiao et al., 2015)?\n- For ablation study, what happens if you use only one of sample selection or sample relabeling?\n- Tab. 3: balanced selection refers to weighting $p'$ by $\\pi^{-1}$?\n\n\n**Minor comments and typos**\n- Section 3.3 \"are the inverses of the entries of the vector $\\pi$ of the class probabilities in the whole dataset\" is hard to read. I assume you meant something like \"The vector $\\pi$ contains the class probabilities in the whole dataset, and we denote by $\\pi^{-1}$ its elementwise inverse\".\n- Section 4.4 has wrong citation format. It should be \"DivideMix (Li et al., 2020a)\" and not \"DivideMix Li et al., (2020a)\" (there are more of those in this paragraph and possibly elsewhere).\n- Section 4.2 \"Genrally->Generally\"\n- Table 7 caption \"ANIMAl->ANIMAL\"\n\n\n**Post-rebuttal/revision**\nThe authors have addressed many concerns, especially experiment-wise. New comparisons were added (C2D still missing from Tab. 6 though -- the results for WebVision appear in table 4 in the paper), as well as Clothing1M experiments. The presentation still can be improved, in particular, I suggest to proofread the newly-written part.\n\n**References**\n\nNishi, Kento, et al. \"Augmentation strategies for learning with noisy labels.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nXiao, Tong, et al. \"Learning from massive noisy labeled data for image classification.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.\n\nXu, Youjiang, et al. \"Faster meta update strategy for noise-robust deep learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nZheltonozhskii, Evgenii, et al. \"Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels.\" arXiv preprint arXiv:2103.13646 (2021).", "summary_of_the_paper": "The paper proposes a two-stage approach to learning with noisy labels (LNL).\n1. a. Clean sample selection based on cosine similarity with k nearest neighbors in embedding space: the average of class distribution of those neighbors should be consistent with the label for sample to be selected.  \n    b. Noisy sample relabeling using temporal self-ensemble: the prediction is defined is average over last L epochs. The sample is relabeled if the confidence of the prediction is higher than some threshold.\n2. Training with self-consistency regularization: the regular training with mixup regularization is performed on selected and relabeled samples, with consistency regularization in form of cosine similarity.\nThe paper tests the performance of method on multiple datasets both with synthetic and real-life noise.", "summary_of_the_review": "The proposed method is novel and interesting and shows promising results in various settings.  My main problem is quality of writing and presentation. if the authors can address my comments and questions and improve the general presentation quality, I'll happily increase the recommendation to accept.\n\n**Post-rebuttal/revision**\nThe authors have done good job addressing my concerns (as well as other reviewers'). The experimental results look interesting and while presentation (in particular writing) can be improved, I think the paper should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YFf5VqIqENs", "reviewer": "Reviewer_pgYs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of this paper is ad-hoc, which stacks the benefits of sample selection and contrastive learning. Considering several previous works have explored them [1, 2, 3] and the proposed method in this paper has minor difference with previous works, it is below the bar of ICLR for the area of learning with noisy labels.\n\n[1] Label Distribution for Learning with Noisy Labels. IJCAI 2020\n[2] Contrastive Learning Improves Model Robustness Under Label Noise. CVPRW 2021\n[3] Learning from Noisy Data with Robust Representation Learning. ICCV 2021", "summary_of_the_paper": "This paper focuses on learning with noisy labels problems, which constructs an iterative learning framework to refine labeling set and train the model parameters.  It demonstrates its advantages by comparing with current baselines on a range of datasets.", "summary_of_the_review": "This paper focuses on learning with noisy labels problems, which constructs an iterative learning framework to refine labeling set and train the model parameters.  It demonstrates its advantages by comparing with current baselines on a range of datasets. However, the novelty of this paper is ad-hoc, which stacks the benefits of sample selection and contrastive learning. Considering several previous works have explored them [1, 2, 3] and the proposed method in this paper has minor difference with previous works, it is below the bar of ICLR for the area of learning with noisy labels. Another problem is k-NN search for neighborhood voting is computational-expensive especially for large-scale datasets.\n\n[1] Label Distribution for Learning with Noisy Labels. IJCAI 2020\n[2] Contrastive Learning Improves Model Robustness Under Label Noise. CVPRW 2021\n[3] Learning from Noisy Data with Robust Representation Learning. ICCV 2021", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "P5nk9qojMia", "reviewer": "Reviewer_cqfk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1.  The paper is easy to follow. The proposed \"S3\"  framework achieves very good performance on both close-set label noise and open-set label noise. \n\n2. The ablation studies are well designed. Each part of S3 along with the hyper-parameters are analyzed. \n\nConcerns:\n\n1. I do not think \"S3\" is a simple framework or simple method. It contains two stages with multiple techniques such as KNN voting, relabelling, sample selection, mixup and consistency loss, which make it hard to understand which part exactly contributes most to the performance. Further, I am worried that the techniques such as KNN voting may increase the training time. Authors are encouraged to calculate the training time of each module in S3. \n\n2. KNN voting-based relabelling, mixup and consistency loss have already been explored in the literature of learning with noisy labels [A1, A2, A3].  I do admire the performance gain brought by S3 in the experiments. However, without deeper analyses, I think the technical novelty is a little limited since it seems that S3 simply combines existing approaches with little modifications. To improve the paper, authors are encouraged to elaborate the main difference among S3 and other approaches in [A1, A2, A3] and state why the combination or the modification is important. \n\n3.  From Figure 2 (a), the network still has good performance when $\\theta_{r}$ approximates 1. Does it mean that relabelling is not very necessary? I think it will be much better if authors calculate the F-score to represent the quality of sample selection and relabelling especially when open-set label noise involved. \n\n\n\nReference:\n\nA1: Multi-Objective Interpolation Training for Robustness to Label Noise. CVPR 2021\n\nA2: Unsupervised Label Noise Modeling and Loss Correction. ICML 2019\n\nA3: Consistency Regularization Can Improve Robustness to Label Noise. ICML 2021\n\n", "summary_of_the_paper": "This paper proposes \"S3\" framework for learning with noisy labels. Specifically, S3 consists of two stages. In the first stage, a relabelling approach and normalized neighboring voting are utilized to guide efficient sample selection; in the second stage, supervised loss (Mixup) and self-consistency loss are used to train networks on selected samples. S3 can be applied to both close-set label noise and open-set label noise and exhibit good performance on several benchmark datasets. ", "summary_of_the_review": "**Pre-rebuttal:** S3 achieves very good performance but the technical novelty is limited. In the current version, it seems that S3 is more like a combination work. Thus I give the initial score of 5. I will consider increasing my score if authors can well clear my concerns.\n\n\n**Post-rebuttal & Final recommendation:**\n\nI thank the authors for their further reply and respect to other works. \n\nIt is relatively a hard decision for me. Experimentally, this paper does achieve promising performance. The authors did a good literature review in the paper \nand elaborate the difference among S3 and other methods in the rebuttal. The designed S3 framework only has few hyper-parameters which are shown robust. To this, I appreciate the efforts and contribution.  However, I still think the combination of components of S3 lacks enough novelty since each component is not new in the literature. \nFor the DivideMix that the authors mention, I admit that DivideMix is also a combination paper. However, the performance of DivideMix \nconsistently outperforms all the previous methods by a large margin especially when the noise rate is high (at that time). While it seems S3 only has little improvement compared to C2D, which is also a work that uses self-supervised learning. In some cases, C2D is even better. Further, considering the supervised, self-supervised training framework is also explored in other works (Co-learning) and S3 currently does not have theoretical justifications such as why KNN voting-based relabelling, or other component, works on both close-set and open-set label noise, I remain my final score as 5.  \n\n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work describes a two-stage method for learning with noisy labels. The crux of the reviews, discussions with the authors and post-rebuttal discussions between reviews (and myself) was related to the novelty of this work. The main concern is that while this body of work presents a relatively solid method (from an empirical point of view), the underlying components are not altogether that novel, and have been used in the context of learning with noisy labels before. Fundamentally, the proposed S3 method did not feel *convincingly* better, given its relative lack of novel technical insights. I appreciate that this is a frustrating reasoning to get -- after all, much of what we do in empirical ML is combinations of existing things. Ultimately, there was consensus amongst the reviewers that the work did not have sufficient insights or such outstanding empirical results so as to overcome this relative lack of technical novelty. 

All the reviewers have engaged meaningfully in discussions, provided constructive feedback and I hope that this will make subsequent iterations of this work better in many dimensions.
Decision: Reject