SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning
Authors: ZHUOWEI WANG
Jing Jiang
Bo Han
Lei Feng
Bo An
Gang Niu
Guodong Long
Venue: Accepted by TMLR ()

Abstract: Deep learning with noisy labels is a challenging task, which has received much attention from the machine learning and computer vision communities. Recent prominent methods that build on a specific sample selection (SS) strategy and a specific semi-supervised learning (SSL) model achieved state-of-the-art performance. Intuitively, better performance could be achieved if stronger SS strategies and SSL models are employed. Following this intuition, one might easily derive various effective noisy-label learning methods using different combinations of SS strategies and SSL models, which is, however, simply reinventing the wheel in essence. To prevent this problem, we propose SemiNLL, a versatile framework that investigates how to naturally combine different SS and SSL components based on their effects and efficiencies. We conduct a systematic and detailed analysis of the combinations of possible components based on our framework. Our framework can absorb various SS strategies and SSL backbones, utilizing their power to achieve promising performance. The instantiations of our framework demonstrate substantial improvements over state-of-the-art methods on benchmark-simulated and real-world datasets with noisy labels.


Published in Transactions on Machine Learning Research (07/2022)
SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning
Zhuowei Wang Jing Jiang! Bo Han Bo An Gang Niu Guodong Long'
Lei Feng
University of Technology Sydney "Honn Kong Baplist Universiiy Chongginng University
NNngaang Technological University zhuowei, wang@stadent. uts. edu, on, fing.fangg@ee ada blarmm@coomp. hubba, alsb..edu.hal edu.hk, Yeng@ecqu.ede edu. boon@ntu, edu.sg, gang. nuu. ml inlggmalllcom com, guodong. goodong.lnng@tin long@uts edu..ox
*RIKEN Center for Advanced Intellignnce Project
Reviewed OpenReview: https: openreview. net/ net/ id LIyg
Abstract
Deep learning with noisy labels is challenging task, which has received much attention from the machine learning and computer vision communities. Recent prominent methods that build on a specific sample selection (SS) strategy and specific semi-supervised learning (SSL) model achieved state-of-the-art performance. Intuitively, better performance could be achieved stronger SS strategics and SSL models are employed. Following this intuition, one might, easily derive various effective noisy-label learning methods using different combi- nations of SS strategies and SSL models, which is, however, simply reinventing the wheel in essence. To prevent this problem, propose SemiNLL, versatile framework that investi- gates how to naturally combine different SS and SSL components based their effects and efficiencies. We conduct systematic and detailed analysis of the combinations of possible components based on our framework. Our framework can absorb various SS strategies and SSL backbones, utilizing their power to achieve promising performance. The instantiations of our framework demonstrate substantial improvements over state-of-the-an methods on benchmark-simule and real-world datascts with noisy labels.
1 Introduction
Deep Neural Networks (DNNs) have achieved great success in various real-world applications, such as image classification (Krizhevsky et al., 2012), detection (Ren et al., 2015), semantic segmentation (Long et al. 2015), anomaly detection (Liu et al. (021), and reinforcement learning (Yu al. 2020 Yang al. 2021 2022). Such a great success is demanding for large datasets with clean human-annotaled labels. However, it costly and time-consuming to correctly label massive images for building large-scale dataset like ImageNet (Deng et al. 2009). Some common and less expensive ways to collect large datasets are through online search engines (Schroff et al.) 2010 or crowdsourcing Yuu et al. 22018). which would, unfortunately, bring wrongly annotated labels to the collected datasets. Besides, an in-depth study (Zhang et al. 2016
showed that deep learning with noisy labels can lead to severe performance deterioration. Thus, crucial
to alleviate the negative effects caused by noisy labels for training DNNs.
A typical strategy is to conduct sample selection (SS) and to train DNNs with selected samples (Han et al. 2018 Jiang al. 2018 Song et al. 2019; Yu et al., 2019 Wei et al., 2020 Yao et al. 2020a Xia et al. 2022 Since DNNs tend to learn simple palterns first before fitting noisy samples (Arpit el al. 2017)) many studies utilize the small-loss trick, where the samples with smaller losses are taken as clean ones. For example, Co-teaching (Han et al. 2018) leverages two networks to select small-loss samples within each mini-batch for training each other. Later, Yu et al. (2099) pointed out the importance of the disagroement
*Corresponding author: Lei Feng.

---

Published in Transactions on Machine Learning Research (07/2022)
between two networks and proposed Co-teaching+, which updates the two networks using the data on which the two networks hold different predictions. By contrast, JoCoR (Wei et al., 2020) proposes to reduce the diversity between two networks by training them simultaneously with a joint loss calculated from the selected small-loss samples. Although these methods have achieved satisfactory performance by training with selected small-loss samples, they simply discard other large-loss samples which may contain polentially useful information for the training process.
To make full use of all given samples, prominent strategy is to consider selected samples labeled "clean" " data and other samples as unlabeled data, and to perform semt-supervised learning (SSL) (Laine & Aila
2016
Tarvainen & Valpola, 2017. Berthelot et al.; 2019, Arazo et al., 2020 Xie ct al., 2020 Sohn et al.
2020 Zhang et al.) 2021; Liu et al. 2022). Following this strategy, SELF Nguuen et al. 2020) detects clean samples by removing noisy samples whose eelfmbn predictions of the model do not match the given labels in each iteration. With the selected labeled and unlabeled data, the problem becomes an SSL, problem, and Mean- -eann-e model (Tarvainen Valpola, 2017) can be trained. Another recent method, DivideMMrr (Li et. al. 2020b), leverages Gaussian Mixture Model (GMM) (Permuter al. 2006) to distinguish clean and noisy data. After removing the labels of noisy samples, DivideMir uses a strong SSL backbone called MixMatch [Berthelot et al. 2019) for training.
As shown above, both methods rely on a specific SS strategy and a specific SST model. The two components play a vitally important role for combating label noise, and stronger components are expected to achieve better performance. This motivates us to investigate a versatile algorithmic framework that can leverage various SS strategies and SSL models. In this paper, we propose SemiNLL, which is a versatile framework to bridge the gap between SSL and noisy-label learning (NLL). Our framework can absorb various SS strategies and SSL backbones, utilizing their power to achieve promising performance. Guided by our framework, one can easily instantiate a specific learning algorithm for NLL, by specifying a commonly used SSL backbone with an SS strategy. The key contributions of our paper can be summarized as follows: (i) Our framework can not only provide an important prototype in the NLL community for further exploration into SS and SSL, but can also act as a conclusive work to prevent future researchers from simply reinventing the wheel. (ii) To instantiate our framework, we propose DivideMix+ by replacing the epoch-level selection strategy of DivideMir (Li et al. 2020b) with mini-batch level one. We also propose GPL, another instantiation of our framework that leverages two-component Gaussian misture model [Li et al.
2020b Permuter et al. 2006)
to select labeled (unlabeled) data and uses Pseudo-Labelinn (Arazo et al.. 2020) the SST. backbone. (iii) We conduct extensive experiments on smnchmark-simulltter and real-world datasets with noisy labels. Our instantiations, DivideMiz+ and GPL, outperform other state-of-tho-art noisy-label learning mothods. We
also analyze the effects and efficiencies of different instantiations of our framework.
The rest of this paper is organized as follows. In Section 2 we first review the related works. Then, the overview of the framework is introduced in Section Section @illsttrate the instantiations of our framework in detail. After that, demonstrate the experimental results in Section and give detailed ablation studies and discussions in Section G
The conclusion is in Section 7
2 RELATED WORK
In this section, we briefly review several related aspects on which our framework builds.
2.1 Learning with noisy labels
For NLL, most of the existing methods could be roughly categorized into the following groups:
Sample selection. This family of methods regards samples with small loss as "clean" and trains the model only on selected clean samples. For example, self-paced MentorNet (Jiang et al. 2018), or equivalently self-teaching, selects small-loss samples and uses them to train the network by itself. To alleviate the sample-selection bias in self- seff-teaching, Han et al. (0118) proposed an algorithm called Co-teaching, where two networks choose the next batch of data for each other for training based on the samples with smaller loss values. Co-teaching+ (Yu et al. 2019) bridges the disagreement strategy (Malach & Shalev-Shwartz 2017) with Co-leaching (Han et. al.) 2018 by updating the networks over data where two networks make different
2

---

Published in Transactions on Machine Learning Research (07/2022)
predictions. In contrast, Wei et al. (2020) leveraged the agreement maximization algorithm (Kumar et al, 2010) by designing joint loss to train two networks on the same mini-batch data and selected small-loss samples to update the parameters of both networks. The mini-batch SS strategy in our framework belongs to this direction. However, instead of ignoring the large-loss unclean samples, we just discard their labels and exploit the associated images an SST. setup.
Noise transition estimation. Another line of NLL is to estimate the noise transition matrix for loss correction (Natarajan et al., 2013 Menon et al. 2015 Xiao et al.; 2015 Goldberger & Ben- Ben-Reuven, 2016:
2017: Hendrycks et al., 2018 Wang et al. 2020 Yao et al. 2020b Wu et al.) 2021a). Patrini ct al. 2017) first estimated the noise transition matrix and trained the network with two different loss corrections. Hendrycks et al.] (2018) proposed a loss correction technique that utilizes a small portion of
Patrini et al.
trusted samples to estimate the noise transition matrix. Wang et al. (2020) proposed a model-agnostic approach to learn the transition matrix directly from data via meta-learning. However, the limitation of these methods is that they do not perform well on datasets with large number of classes.
Other deep learning methods. Some other interesting and promising directions for NLL include meta- learning (Finn et al.
2017 Snell et al. 2017 Li et al. 2020a) based, pseudo-label estimation (Lee) 2013
2017: Ma et al., 2020 Wang et al. 2019 Xu et al. 2019 Zhang & Sabuncu 2018: Hu et al. 2021) | based approaches. For meta-learning based approaches, most
based, and robust loss (Feng et al. |020 Ghosh et al.
studies fall into two main categories: training a model that adapts fast to different learning tasks without overfitting to corrupted labels (Garcia et al.) 2016 Li et al., 2011)), and leurning to reweight loss of each
mini-batch to alleviate the adverse effects of corrupted labels (Ren et al. 2018; Shu et al.] 2019 Zhang
et al. 2020 Wu et al. 2021b; Zhheng et al., 2021 Pseudo-label estimation based approaches reassign the
labels for noisy samples. For example, Loint-Optim
(Tanaka et al. 2018 corrects labels during training and
updates network parameters simultaneously. The family of pseudo-label estimation has a close relationship
with semi-superrvied learning (Han et al.) 2019 Lee 2013; Tanaka et al. 2018 Yi & Wu 2019). Robust
loss based approachos focus on designing loss functions that are robust to noisy labels. Recently, some
works (Wang el al. 2022 Xu el. al.] 2022) non-trivially extend NLL to federated setting
MMcMahan el al.
2017 Jiang et al. (020). which is a distributed environment with non-IID data (Ma et al. 2022] Chen et al.
2022
privacy-preserving (Long et al. 2020 2022), and heterogencous model architecture
[Tan et al. 2021
across clients or devices.
2.2
Semi-supervised learning
SSL methods leverage unlabeled data to provide additional information for the training model. A line of work is based on the concept of consistency regularization: if perturbation is given to unlabeled sample, the model predictions of the same sample should not. be too different. Laine & Aila (2016) applied consistency between the output of the current network and the exponential moving average (EMA) of the output from the past epochs. Instead of averaging the model outputs, Tarvainen & Valpola (2017) proposed to update the network on every bati- using an EMA of model parameter values. [Brtthlot et al. (2019) introduced a holistic approach that well combines MixUp (Zhang et al., 2018), entropy minimization, and consistency regularization. Another line of SSL is pseudo-labeling, the objective of which is to generate pseudo-labels for unlabeled samples to enhance the learning process. Arazo et al. (2020) proposed a method to improve previous pseudo-labeling methods (Iscen et al.] 2019) by adding MizUp augmentation (Zhang et al. 2018). Xic et al. (2020) and [Sohn et al. (2020) used confidence-basee strategy pscudo labeling to select unlabeled data with high confidence. [Zhang el al. 2021 proposed a curriculum learning method to leverage unlabeled data for SSL,
2.3 Combination of SS and SSL
Some previous studies that combine a specific SS stralegy and a specific SSL backbone could be regarded as special cases in our framework. Ding et al. (2018) used a pre-trained DNN on the noisy dataset to select labeled samples. In the SSL stage, Temporal Ensembling (Laine & Aila, 2016) was used to handle labeled and unlabeled data. Nguyen et al. (2020) proposed a progressive noise filtering mechanism based on the Mean- Teacher model (Tarvainen & Valpola 2017) and its self-ensemble prediction. Li et al. (2020b) used
3

---

Published in Transactions on Machine Learning Research (07/202) 2022)
strategy
SSL
Noisy set
Mini-batch n-1
Fordwwar
Labeled set
Unlabeled
Classified set
Labeled samples
samples
Mini-batch
Formau
strategy
Backwarrd
Network
CDC
The schematic SemiNL.L First, each mini-batch of data forwarded to the network conduct SS, which divides the original data into the labeled/ abbledd/nlabeele uunlabeled sets. Second, labeled/unlabeled /nllabeled samples are used to train the SSL backbone to produce accurate model output.
Figure 1:
Gaussian Mixture Model (GMM) to divide noisy and clean samples based on their training losses and fitted them into recent. SSL algorithm called MixMalch (Berthelot et al., 2019). Each specific component used in these methods has its own pros and cons. This motivates us to propose a versatile framework that can build on various strategies and SSL backbones. In other words, many recent publications (Arazo et al, 2019 Li al. 22020b Nguyen et al. (2000) preprints (Cordeiro et al. 2021 Wei al.) (021b) could be taken as special instantiations of our framework, which indicates that conclusive work like this paper is vitally nccessary prevent futurre rescarchers from simply reinventing the wheel.
3 The Overview of SemiNLL
In this section, we present SemiNLL, a versatile framework of learning with noisy labels by SSL. The idca behind our framework that we effectively take advantage of the whole training set by trusting the labels of undoubtedly correct samples and utiliz- ing only the image content of potentially corrupted samples. Previous sample selection methods Han
Algorithm 1 SemiNLL Input:
Network fe, SS strategy SELECT, SSL method SEMI, epoch T Tmax, iteration Imaxi
for / 1,2....,Tuue do
2: Shuffle training set Dirains
3:
for 1,..., Imax do
4: Fetch mini-batel Dn from Diraini
let al. et al.
2018 Jiang et al. 2018 Yu et al., 2019 Wei
5: Obtain Xm, Um + SELECT (Dn.
2020 train the network only with selected
6: Update fo +- SEMI(X Um, fo);
clean samples, and they discard all potentially cor-
rupted samples to avoid the harmful memorization of DNNs caused by the noisy labels of these samples. In this way, the feature information contained in the associated images might be discarded without being exploited. Our framework, alternatively, makes use those corrupted samples by ignoring their labels while keeping the associated image content, transforming the NLL problem into an SSL setup. The mechanism of SSL that leverages labeled data guide the learning of unlabeled data naturally fits well in training the model with the clean and noisy samples divided by our SS strategy. We first discuss the advantages of the mini-batch SS strategy in our framework and then introduce several SSL backbones used in our framework. The schematic of our framework is shown in Figure]
7:
end for
8: end for
Output:
3.1 Mini-batch sample selection
During the SS process, a hazard called confirmation bias (Tavvainen & Valpola 2017) is worth noting, Since the model is trained using the selocted clean (labeled) and noisy (unlabeled) samples, wrongly selected clean samples in this iteration may keep being considered clean ones in the next iteration due the model overfitting to their labels. Most existing methods (Li et al. 2020b Nguyen al. 2020 divide the whole
4

---

Published in Transactions on Machine Learning Research (07/2022)
training set into the clean/noisy set epoch level. Our mini-batch SS strategy divides each mini- batch of samples into the clean subset X., and the noisy subset Um (Line 5 in Algorithm I right before updating the network using SSL backbones. The advantages of our mini-batch SS over epoch-wise SS are: (i) Avoiding confirmation bias: In the case of epoch-wise SS, the divided clean/noisy sets are incorporated into the SSL phase and will not be updated till the next epoch. Thus, the confirmation bias induced from those wrongly divided samples will accumulate within the whole epoch. Our mini-batch SS strategy divides each mini-batch of samples into clean/noisy batches before updating the network using SSL backbones. In the next mini-batchi the updated network can better distinguish clean and noisy samples, alleviating the confirmation bias mini-batch by mini-batch. (ii) Improving computational efficiency: Since the time complexity of most SS methods is not linear, the number of operations increases dramatically as the input size increases. Table 6.4 compares the training time of DividMix+ (mini-balch-wise) and DivideMix (epoch- wise) on CIFAR- 10, showing DivideMix+ is more computationally efficient than DivideMix in both the ss process and the whole training process. (iii) Injecting stochasticity: During the epoch-wise SS process, the model tends to select the confident samples that have been selected in previous epochs due to the model overfitting their labels. In this way, some confident but noisy samples will keep being selected by the model, resulting in performance degradation. The mini-batch-wir wist SS can inject stochasticity in training since each mini-batch of data is randomly sampled from the whole dataset, avoiding the model constantly selecting the same confident samples. (iv) Up-to-date model for SS: The mini-batch-wise usage of SS and SSL, makes the data selection up-to-date. The model used to select the clean samples is updated using the SSL method at each mini- batch.
3.2 SSL backbones
The mechanism of SSL that uses labeled data to guide the learning of unlabeled data fits well when dealing with dlean/noisy data in NLL. The difference lies in an extra procedure, as introduced in Subsection 3.1 that divides the whole dataset into clean and noisy data. After the SS process, clean samples are considered labeled data and keep their annotated labels. The others are considered noisy samples, and their labels are discarded to be treated as unlabeled ones in SST. backbones. SemaNT. can build on a variety of SSL algorithms without any modifications to form an end-to-end training scheme for NLL. Concretely, we consider the following representative SSL backbones ranging from weak to strong according to their performance in SSL tasks: (i) Temporal Ensembling (Laine& & Aila 2016). The model uses an exponential moving average (EMA) of label predictions from the past epochs as a target for the unsupervised loss. It enforces consistency of predictions by minimizing the difference between the current outputs and the EMA outputs. (ii) MixMalch [Berthelot et al. 2019). MixMatcc is a holistic method that combines MizUp (Zhang et al. 2018), entropy minimization, consistency regularization, and other traditional regularization tricks. It guesses low-entropy labels for augmented unlabcled samples and mixes labeled and unlabeled data using MizUp (Zhang et al. 2018). (iii) Pseudo-Tabeling (Arazo et al. 2020). This method learns from unlabeled data by combining soft psewdo-label generation (Taakaa et al.) 2018) and MirUp augmentation (Zhang et al. 2018) to reduce confirmation bias in training. In the next section, we will instantiate our framework by applying specific SS strategies and SSL backbones to the SELECT and SEMI placeholders in Algorithm
4
The Instantiations of SemiNLL
4.1 Instantiation 1: DivideMix+
In Algorithm [ if we (i) specify the SELECT placeholder a GMM (eermueer et al. 2006 (ii) specify the SEMI placeholder MizMatch (Berthelot et al. 2019) mentioned in Subsection 3.2 and (ii) train two independent networks wherein each network selects clean/noisy samples in the SS phase and predicts labels in the SSL phase for the other network, then our framework is instantiated into mini-batch version of Dir (Li el al. (2020b). Specifically, during the SS process, DivideMia (Li et al.) 2020b) fits a two- component GMM to the loss of each sample and obtains the posterior probability of a sample being clean or noisy. Since DNNs tend to learn simple patterns first before fitting noisy samples (Arpit et al.) 2017). clean/ noisy samples have lower/ higher loss during the early epochs of training. Therefore, we can determine whether sample is more likely to be clean or noisy based on its loss value. Given classification loss &
5

---

Published in Transactions on Machine Learning Research (07 (07/2022)
(e.g., cross entropy) of sample (1) in the dataset, calculate the probability density function (pdf) of a mixture model of & (in our case k = 2) components on the loss & using Expectation-Maximiaatai technique:
p(k) p(e, k)
p(k 4)) p((;)
(1)
where k- = (1) denotes the clean (noisy) set, and p(clean ()) is the posterior of the smaller-mean (clean) component of the GMM. The clean set Xe and the noisy set Ue can be defined as
X. = Jt) :p (clean () Ue = ((x,, (clean [+)<< <},
(2)
where r is a threshold on p(cloan 4.. During the SSL phase, the clean set X. and the noisy set Ue are fit into an improved MixMalch (Berthelot et al.) 2019) strategy with label co-refinement and co-guessing. As shown in Figure 4 (b) in Appendix, the SS strategy (GMM) of DivideMir (Li et al., (020b) is conducted an epoch level. Since X. and Ue are updated only once per epoch, the confirmation bias induced from the wrongly divided samples will be accumulated within the whole epoch. However, our mini-batch version, which is called DinideMia++ (Figure E (c) in Appendix), divides each mini-batch of data into a clean subset Xm and noisy subset Um. and updates the networks using the SSL backbone right afterwards. In the next mini-batch, the updated networks could better distinguish clean and noisy samples.
4.2 Instantiation 2: GPL
Intuitively, the choice of stronger SS strategics and SSL models would achieve better performance based on our framework. Thus, we still choose GMM to distinguish clean and noisy samples due to its flexibility in the sharpness of distribution (Li et al.] 2020b). As for the SSL backbone, we choose the strongest Sgeudo-Labeling (Arazo et al.] 2020 introduced in Subsoction 3.2. We call this instantiation GPL (GMM + Pseudo Labelig-. To keep our instantiation simple, we do nol train two networks in GPL as in DivideMir (Li et al. 2020b) and Divide To our understanding, training two networks simultancously might provide significant performance improvements. We will give detailed discussions about the effectiveness and efficiency
of two networks in Section 6.5
4.3 Self- Self- prediction divider
Inspired by SELF (Nguyen et al. 2020 we introduce the peee-prediition divider, simple yet effective SS strategy which leverages the information provided by the network's 's own prediction to distinguish clean and noisy samples. Based on the phenomenon that DNN's predictions tend to be consistent on clean samples and inconsistent on noisy samples in different training iterations, we select the correctly annotated samples via the consistency between the original label set and the model's own predictions. In each mini-batch of data, if the model predicts the class with the highest likelihood to be the annotated label, the sample (x;, V:) is divided into the clean set Xm. Otherwise, the sample is divided into the noisy set Um- To be more specific, the model prediction ý; is defined as
8, A arg max .cy (x;
(3)
where C' is the number of class, 0 is the model parameters, and f(x;; 0) is the softmax output of the model. The clean set Xm and the noisy set Um can be defined as
Xm = (x). -.), Um = ((x)))) 9 * Fc).
Compared to previous small-loss SS methods (Han et al. 2018 Wei et al. 2020 Yu et al. 2019), which depend on known noise ratio to control how many small-loss samples should be selected in cach training iteration, self-prediction divider does not need any additional information to perform SS strategy where the clean subset and the noisy subset are determined by the network itself. Concretely, we instantiate three learning algorithms by combining our self- pefe-rreiiction divider (SPD) with three SSL backbones introduced in Subsection 3.2 and denote them as SPD- Temporal Ensembling. MPt-MirrMatc,, and SPD-Pseudo Pebbb-ddd--.. respectively.
6

---

Published in Transactions on Machine Learning Research (07/ 2202)
4.4 Effects of the two components
This section demonstrates the effects of SS strategies and SSL backbones in our framework. To prove that a more robust SS strategy can boost performance for our framework, we propose DivideMix- (Figure 4 (a) by replacing the GMM in DivideMir (Li et al. 2020b) with our self-prediction divider on an epoch level. Since relf-prediction diwider is supposod to be weaker than GMM, DivideMiz- expected to achieve lower performance than DivideMir (Li et al.; 2020b). To prove the effectiveness of the SSL backbone, we remove it after the SS process and only update the model using the supervised loss calculated from the clean samples. We will give detailed discussions in Section 0
5 Experiment
5.1 Experiment setup
Datasets. We compare our method with five state-of-the-art algorithms in PU learning: We thor- oughly evaluate our proposed DivideMia+ and GPL on six datasets, including MNIST (Lecun al. 1998), FASHION-MNIST (Xiao et al.) 2017). CIFAR-10, CIFAR-100 (Krizhevsky & Hinton 2009), Cloth- ingIM [iioo et al, 2015), and WebVision (Li et al. 2017). The detailed characteristics of the datasets in the experiments are shown in Table MNIST and FASHION-MNIST contain 60K training images and 10K test images of size 28 x 28. CIFAR-10 and CIFAR- contain 50K training images and 10K test images of size 32 x 32 with three channels. According to previous studies (Zhang & Sabuncu 2018 Li et al.) 2020b; Wei et al. 2020); we experiment with two types of label noisc: symmetric noise and asymmetric noise. Sym- metric label noise is produced by changing the original label to all possible labels randomly and uniformly according to the noise ratio. Asymmetric label noise is similar to real-world noise, where labels are flipped to similar classes. ClothingIM is large-scale real-world dataset that consists of one million training images from online shopping websites with labels annotated from surrouniing texts. The estimated noise ratio is about 40%. WebVision is a large-scale dataset which consists of real-world web noise. We follow Chen el al. (2019) Li et al. (020b) to create a mini version of WebVision that uses the Google subset images of the top 50 classes.
Network structure and optimizer. Following previous works (Arazo et al. 2020 Li 2020b Wei ct al.) 2020 Zhang & Sabuncu, 2018), use 2-layer MLP for MNIST, a ResNet-18 He et al., 2016)) for PASHION-MNIST, the well-known "13-CNN" architecture (Tarvainen & Valpola, 2017) for CIFAR-10
and 10FAR-100, a ResNet-50 (He et al. 2016) for ClothingIM and the inception- v2 (Szegedy et al. 2017) for mini WebVision. To ensure a fair comparison between the instantiations of our framework and other methods, we keep the training settings for MNIST, CIFAR-10, CIFAR-104, Clothing and mini WibVision as close as possible to DivideMMrr (Li el al. 2020b) and FASHION-MNIST close to GCE (Zhang) & Sabuneu 2018). For FASHION-MIITT the network is trained using stochastic gradient descent (SGD) with 0.9 momentum and weight decay of 1 x 10-4 for 120 epochs. For MNIST, CIFAR-10, and CIFAR-100, all networks are trained using SGD with 0.9 momentum and a weight decay of 5 x 10-4 for 300 epochs.
Baselines. We compare DivideMia+ and GPL to previous state-of-the-art algorithms from Co- teaching (Han et al.] correction AArzo et al. 2019 and DivideMiz (Li et al.] 2020b). We implement all methods by PyTorch on
2018). F-correction (Patrini et al. 2017), GCE (Zhang & Sabuncu] 2018), M-
NVIDIA Tesla V100 GPUs. (1) Cross-Entropy, which trains the network using the cross-entropy loss. (2) Coteaching (Han et al.) 2018), which trains two networks and cross-updates the parameters of peer networks. (3) F-correction (Patrini et al.
2017), which corrects the prediction by the label transition matrix. As sug- gested by the authors, we first train standard network using the cross-entropy loss to estimate the transition matrix. (4) JoCoR (Wei et al.] 2020), which uses a joint loss to train two networks on the same mini-batch data and selects small-loss samples to update the networks. (5) GCE (Zhang & Sabuncu 2018), which uses a theoretically grounded and easy-to-use loss function, the E, loss, for NLL. (6) M-correction (Arazo et al. 2019), which models clean and noisy samples by fitting a two-component BMM and applics MixUp data augmentation (Zhang et al. 20188). (7) DivideMie (Li et al. 2020b), which divides clean and noisy samples by using GMM on an epoch level and leverages MiaMatth (Berthelot et al. 2019) as the SSL backbone.
7

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Table 1: Average test accuracy (%) and standard deviation (5 runs) various datasets under symmetric label noise The best accuracy is bold-faced. The socond-best accuracy is underlinod.
Symmetric Noise ratio
Datasets
Method
Mean
20%
Cross-Entropy
86.16 + 0.31
70.39 + 0.59
50.35 = 0.51
23.41 = 0.96
57.58
Co-teaching
91.20 0.03
90.02 + 0.02
83.21 = 0.71
25.33 = 0.84
72.44
R-correction
93.93 +: 0.10
84.30 +: 0.43
65.06 = 0.64
29.81 0.63
68.27
JoColt
94.30 +: 0.09
92.66 + 0.13
89.94 0.24
75.37 + 0.74
88.07
MNIST
GCE
94.36 +: 0.11
93.61 + 0.17
92.46 = 0.20
85.04 = 0.66
91.37
M-correction
97.25 + 0.03
96.63 + 0.04
95.07 = 0.08
86,19 = 0.42
93.79
DivideMix
96.80 + 0.08
96.53 = 0.06
96.47 = 0.04
95.15 + 0.25
96.24
GPL (ours)
96.67 + 0.09
96.27 + 0.08
95,82 = 0.09
94.81 + 0.15
95.89
DivideMix+ (ours)
96.83 + 0.06
96.79 + 0.06
96.69 + 0.03
95.91 + 0.10
96.56
Cross-Entropy
90.83 1: 0.26
86.44 1: 0.11
77.27 0.56
61.84 1: 1.27
79.10
Co-tcaching
89.18 0.32
89.13 0.05
80.08 0.25
60.36 :: 2.15
79.69
F-correction
93.37 + 0.17
92.27 1: 0.06
90.32 0.30
85. + 0.06
90.43
FASHION MNIST
JJCoR
91.43 0.14
90.55
$6.89 0.29
79.61 0.11
87.12
GCE
93.35 +: 0.09
92.58 +: 0.11
91.30 0.20
88.01 + 0.22
91.31
M-correction
93.03 0.15
92.74 0.42
91.61 0.02
85.25 1: 0.23
90.66
DivideMix
92.98 1 0.17
92.55 1: 0.13
91.55 0.31
88.55 0.24
90.66
GPL (ours)
91.38
87.14 +: 0.65
DivideMix+ (ours)
92.89 0.15
92.15 1 0.16
88.70
91.74
Cross-Entropy
83.48 + 0.17
68.49 + 0.40
45.65 = 0.06
27.56 + 0.43
57.05
Co-traching
67.73 + 0.71
62.83 + 0.72
48.81 + 0.78
27.56 + 2.71
51.73
P-correction
83:27 +: 0.04
73.67 +: 0.30
77.64 = 0.11
63.95 + 0.32
74.63
JoCoR
85.22 0.06
80.27 1: 0.37
58.72 = 0.29
29.67 + 0.68
63.47
CIFAR-10
GCE
87.75 + 0.05
84.11 = 0.26
72.84 + 0.30
83.61
M-correction
92.01 :: 0.40
90.09 + 0.68
85,90 = 0.22
70.57 + 0.85
84.64
DivideMix
94.82 + 0.09
93.95 + 0.14
92.28 0.08
89.30 + 0.17
92.59
GPL (ours)
94.45 +: 0.20
94.00 +: 0.22
93.32 1 0.10
91.76 + 0.23
93.38
DivideMix (ours)
94.84 + 0.12
94.03 + 0.20
93.08 = 0.19
91.91 + 0.07
93.47
CrossEntrropy
60.93 +: 0.40
46.24 +: 0.74
29.00 0.38
11.42 + 0.19
36.90
F-correction
60.49 0.29
48.93 1: 0.21
48.74 0.41
22.93 + 0.78
45.27
JoColt
65.89 0.08
49.65 1 0.51
32.38 0.60
16.91 0.63
41.21
CIFAR-100
GCE
69.20 + 0.10
65.90 + 0.25
57.33 0.18
18.19 + 1.15
52.66
M-correction
67.96 + 0.17
64.48 + 0.76
55.37 0.72
24.21 + 1.06
53.01
DivideMix
73.17 +: 0.28
71.01 +: 0.16
66.61 0.18
43.25 4 0.82
63.51
GPL (ours)
0.24
0.63
59.96
66.47
DivideMix+ (ours)
73.22 + 0.21
71.08
67.52 + 0.19
67.46
5.2 Performance comparison
The results of all the methods under symmetric and asymmetric noise types MNIST, FASHION-MNIST. CIFAR-10, and CIFAR-100 are shown in Table Table The results on Clothing 1 and mini WebVision are shown in Table 3 and Table g. Furthermore, We delve into the reasons beyond these results in Section 6
Results on MNIST. DivideMiz+ surpasses DivideMir across symmetric and asymmetric noise all noise ratios, showing the offeetiveness of the mini-batch SS strategy in our framework. GPL performs worse than DivideMix under symmetric noise while outperforming it under symmetric noise. In the cases of Symmetric 20% and 40%, DivideMiix+ and M-correction perform better than the other methods. When it comes to Asymmetric noise, GPL and M-correction demonstrate better performance. However, the performance of M-correction drops dramatically in the more challenging Symmetric 80% case where Diri surpasses all the other a algorithms.
Results on PASHION-MNIST. FASHION-MNIST quite similar MNIST but more complicated. Div+deMir+ still outperforms DivideMix on symmetric and asymmetric noise at all noise ratios. GPL still performs worse than DivideMix under symmetric noise and better under symmetric noise. Under symmetric

---

Published in Transactions on Machine Learning Research (07/2022)
Table 2: Average lest accuracy (%) and standard deviation (5 runs) in various datasets under asymmelric label noise. The best accuracy is bold-faced. The second-best accuracy is underlined.
Asymmetrie Noise ratio
Datasets
Method
Mean
10%
20%
30%
40%
Cross-Entropy
96.78 0.19
91.15
86.01
79.92
88.22
Co-teaching
90.32 0.02
89.03 1 0.02
79.80
0.09
81.02
E-corroction
96.39 0.04
94.27 0.21
0.99
90.40
JoCoR
95.43
94.39
90.15
87.31 0.05
91.82
MNIST
GCE
0.13
0.00
9..1
93.42
94.12
M-correction
9.074
9..0
96.67 0.07
94.55 = 0.40
96.24
DivideMix
96.17
0.00
95.88 - 0.05
95.83 0.05
96.00
GPL (ours)
96.76 0.04
96.71 0.03
96.49 0.08
96.45 0.04
96.60
DivideMix+ (ours)
96.67 0.04
96.66 0.07
96.50 0.04
96.46 0.04
96.37
Cross-Entropy
93.58 + 0.16
92.20 + 0.33
90.11 + 0.67
0.67
81.56 0.41
90.26
Co-teaching
88.01 = 0.03
78.88 + 0.20
70.07 - 0.38
61.97 0.21
74.73
F-correction
94.17 = 0.12
93.88 = 0.10
93.50 + 0.10
93.25 + 0.16
93.7
FASIION MNIST
JoCoR
91.54 + 0.13
88 60 0.47
84.37 0.24
81.68 0.62
86.55
GCE
93.51 + 0.17
93.24 + 0.14
92.21 0.27
89.53 0.53
92.12
M-correction
92.11 1 0.93
91.26 +: 1.35
89.79 + 1.28
89.58 2,20
90.69
DivideMix
91.83 - 0.24
91.09 + 0.08
89.90 + 0.26
87.58 01:88:110 0.26 0.26
90.10
GPL (ours)
92.52 +: 0.22
92.23 1: 0.09
92.15 0.26
91.64 1..44110 0.31
92.14
DivideMix (ours)
92.56 + 0.39
92. + 0.21
91.62 0.08
89.67 0.44
91.53
Cross-Entropy
90.85 0.06
87.23 0.40
81.92 0.32
76.23 0.45
84.06
Co-teaching
62.85 2:20
61.04 1.31
54.50 0.39
51.68 1.66
57.52
F-corroction
80.79 0.33
86.79 0.67
83.34 0.30
76.81 1.08
84.18
JoCoR
88.62 0.21
89.79 0.17
82.37 1
77.90
84.67
CIEAR-10
GCE
90.40 0.09
89.30 0.13
86.89 0.22
82.60 0.17
87.30
M-correction
92.28 0.12
92.13 0.17
91.38 0.11
90.43 0.23
91.56
DivideMix
93.61 + 0.15
92.99 1: 0.21
91.79 0.36
90.37 0.31
92.24
GPL (ours)
0.07
0.06 92.82
DivideMix+ (ours)
94.27 0.23
93.92 0.20
91.91
93.23
Cross-Entropy
68.58 0.34
68.82 1: 0.22
53.99 0.50
44.31 0.23
58.93
E-correction
68.87 + 0.06
64.11 + 0.37
56.45 + 0.59
46.44 : 0.50
58.97
JoCoR
69.44 +
66.91 0.54
54.71 0.42
39.76 0.97
57.71
CIFAR-100
GCE
70.77 + 0.14
0.14
69.22 0.15
64.60 0.25
51.72 1.17
64.08
M-correction
69.44 +: 0.52
67.25 0.81
63.16 1.55
52.90 1.79
63.
DivideMix
74.00 0.29
73.28 1: 0.42
72.84 0.36
54.33 0.69
68.61
GPL (ours)
71.94 + 0.29
71.22 + 0.11
70.56
0.23
69.84 0.41
70.89
DivideMix+ (ours)
73.49 + 0.31
73.30 +: 0.22
72.36
0.43
55.63 0.60
68.70
noise, DivideMix+ outperforms most of other methods, while F-correction and GCE surprisingly achieve comparable test accuracy under asymmetric noise.
Results on CIFAR-10. DivideMia+ constantly out-
Table 3: Test accuracy (%) on Clothing
performs DivideMir, especially in the cases with higher noise ratios. DivideMix+ achieves improvement in the accuracy of +2.61% in Symmetric 80% and +1.34% in Asymmetric 40% over DivideMir. We believe the reason is that the mini-batch SS strategy used in our framework can better mitigate the confirmation bias induced from wrongly divided samples more challenging scenarios. Tn the casiest Symmetric 20%, 40%. DivideMir, DivideMix+ and GPL tie closely with DivideMfrr+ slightly working better than the other two. In the harder cases (symmetric 60%, symmetric 80% and all asymmetric cases), GPL and DivideMix+ surpass the other methods over a large margin. GPL shows superior performance under asymmetric noise.
Methods
Test Accuracy
Cross Ccoss-Ettrpy
69.21
F-correction Patrini
2017
69.81
M-correction
71.00
Joint-Optin
Tanaka el
2018
72.16
Meta- Learning Li ct al.
2019
73.47
PENCIL (Y) Wu
2019
73.19
Dividemix [Li
212
GPLíous)) (ovidemix((ouss)
73.89
74.14
Results on CIFAR-100. There are 100 classes in CIFAR-100, making it more challenging to train than CIFAR-10. Coteaching tends to fail CIFAR-100 even under low noise ratios. most cases, DivideMiz+ and DivideMiz achieve higher test accuracy than the other approaches, with DivideMizz+ performing better. Specifically, DivtdeMia+ surpasses Dir by 14.82% in the hardest. symmetric 80% case. GPL is sur-

---

Published in Transactions on Machine Learning Research (07 (07/2022)
passed by DivideMix in most cases except for the hardest symmetric 80% and asymmetric 40% cases. An interesting phenomenon is that all the approaches suffer from performance deterioration in the asymmetric 40% cases except GPL, which significantly outperforms the second-best algorithm over +14%.
Results on real-world datasets. To verify the robustness of our framework under real-world datasets with noisy labels, we demonstrate the ef- fectiveness of DivideMiz+ and GPL on Clothing1M and (mini) WebVision. As shown in Table 3 the performance of DivideMiz+ is better than that of DivideMir and other methods. GPL is slightly lower than DivideMix without using two networks for training. In Table | both + and GPL. outperform compared methods.
Table 4: Test accuracy (%) on (mini) WibVision.
Methods
Test Accmacy
F-correction
Patrini et al.
2017
61.12
Decoupling Malach & Stulee-Sliwarkz
2017
62.54
D2L (Ma al.
2018
62.68
MentorNet Jang al.
2018
63.00
Co-teaching Han ct
2018
63.58
Therative-CV
65.24
Dividemix et al
2020
77.32
GPL(ours)
77.84
(ovidemix-
78.28
6 Ablation Studies and Discussions
In this section, we investigate our framework in depth to gain more insights. Specifically, we analyze the pros and cons of different SS and SSL, components, and also analyze the effects and efficiencies of the instantiations of our framework, which sheds light on how SSL components affect SS process. Moreover, we conduct an ablation study training with two networks. 1.0
0.5
1.0
0.8
Text Acmmnyy
0.6
Test Acounay
.0.8
Prudsion
Predzion
0.7
0.4
DeMd DiMd GPL
0.4
MivedMMi+
0.5
DiiddMis+
Diddshee
GPL.
GPL
0.2g
200
300
0.5
100
290
300
100
200
300
0.5
100
200
300
Epach
Epoch
(a) CIFAR-10 sym 80%
(b) CIFAR-100 asym 40%
Figure 2: Results of ablation study. For (a) and (b), left: accuracy vs. epochs; right: precision vs. epochs.
6.1 The effects of mini-batch mechanism
In Table 5 the Lest. accuracy of DivideMist+ constantly outperforms DivideMiz in most cases. This is because our mini- batch mechanism injects more stochasticity in training and reduces the harm of confirmation bias. In Figure 2 (a) and (b), the precision of DivideMiz+ outperforms DivideMiz during the training process, showing that our approach is better finding clean instances. We also compare GPL (mini-batch) with GPT (epoch) CIFAR-10 in Appendix D.4
6.2 The effects of different SS strategies
To study how SS strategies can affect the performance of our framework, we propose DivideMix- by re- placing the GMM component in DivideMix with our self-prediction divider yet maintaining the epoch-level SS strategy for fair comparison. In CIFAR-10, the difference between DivideMir- and DivideMir is not obvious in the lower noise ratios. However, in the most difficult symmetric 80% case, the test accuracy of DivideMir is +12.99% higher than DivideMir- and the precision of DivideMiz in Figure 2 (a) outperforms Mir- In 100,R-100, the difference of test accuracy is even greater under symmelric noise. An im- pressive phenomenon to note is that DivideMir- excels in the asymmetric 40% case in CIFAR-100 with the highest precision and accuracy. This because GMM distinguishes clean and noisy samples by fitting their loss distribution, which works effectively for symmetric noise. However, for asymmetric noise, most sam- ples have near-zero normalized loss due to the low entropy predictions of the network, causing performance
10

---

Published in Transactions on Machine Learning Research (07/ (022))
Table 5: Test accuracy (%) of DivideMir,, DivideMir, and DivideMia+.
Symmetric Noise ratio
Asymmettic
Datasets
Method
Noise ratão
20%
40%
60%
80%
10%
20%
30%
10%
DivideMIx-
94.49 0.02
93.64
91.65
0.34
76.61 = 1.26
93.58
00:50000 0.02
02.87
0.14
91:21
0.24
CIFAR-10
DivideMix
91.82 0.09
93.95 = 0.11
92.28 + 0.08
89.30 0.17
93.61 0.15
92.99
0.21
91.79
90.57
9..57
DivideMix+ [ours]
94.84 +: 0.12
94.03
+ 0.20
93.08
+ 0.19
91.91
+ 0.07
94.27 + 0.23
93.92
+ 0.20
92.82
0.28
91.91
0.24
DivideMix-
72.51 0.32
61.13 0.60
25.96 0.78
73.62
0.12
72.32
68.04
CIFAR-100
DivideMix
+3.17
71.01
0.16
06.61 + 0.18
43.25 = 0.82
74.00 + 0.29
73:28
+ 0.42
72.84
0.36
54.33 + 0.00
DivideMix I (aurs)
73.22 1: 0.21 71.03 1: 0.32 67.52 1: 0.19 58.07 1: 0.71
73.49 0.31 73.30
1 0.22
72.36
0.43
55.63 0.60
Table 6: Test accuracy (%) of the baseline and three SSL backbones integrated into our proposed framework.
Dataset
CIFAR-10
CIFAR-100
Method/Noite rate
20%
50%
8U%
20%
50%
80%
Ca--oros-Eatoppa
83.13 = 0.16
79.74 0.10
19.11 = 0.15
15.07 = 0.55
35.02 = 0.57
10:22 0.10
SPP.Temprral Ensembling
83.15 + 0.06
0.36
49.10 0.13
30.91
12.37 0.67
SPD-MixMMath
93.13 0.52
90.22 0.18
88.77 0.20
72.80
0.30
68.57 0.20
33.92 0.20
SPPDPaendo-Tabeling
94.52 +: 0.06
93.24 + 0.36
90.27 1: 0.34
73.84 + 0.48
68.61 + 0.40
55.37 0.34
deterioration for GMM. Since SPD leverages the prediction of its own network to choose clean samples, its performance is more sensilive to noisy rates rather than noise type.
6.3 The effects of different SSL backbones
We evaluate the effects of SSL backbones in our framework by combining the with three different SSL methods and a bascline which only updates the model using the cross-entropy loss caleulated from clean samples. We denote them as SPD- PPD-Temprall Ensembling (TE), SPD-MirMaalh (MM), (PL), and SPDDCrross Cross------trppg Entropy (CE), respectively. For fair comparisons,
divider (SPD)
use the
"I3-CNN" architecture (Tarvainen & Valpola 2017) for all methods across different datasets. We keep most hyperparameters introduced by the SSL methods close to their original papers (Arazo et al. 2020 Berthelot et al. 2019 Laine & Aila 2016), since they can be easily integrated into our framework without major adjustments. In Table 6 SPD-MM and SPD-PL outperform SPD-TE by a large domain in both CIFAR-10
and CIFAR-100, especially under 80% noise ratio. This is reasonable because TE (Laine Aila 2016)
only uses consistency regularization for unsupervised loss, while MM (Berthelot al. 2019) and PL AAazo ct al. 2020) also leverage entropy regularization well as MixUp data augmentation (Zhang et al. 2018).
Moreover, SPD-PL achieves remarkable test accuracy under 80% noise ratio in CIFAR-100. This is due to the additional loss used in SPD-PL that prevents the model from assigning all labels to single class at the early training stage, From the results of SPD-CE, we can see that after the removal of the SSL backbone, the test accuracy drops dramatically compared to SPD-MM and SPD-PL. This is due to the substantial amount of data that has been removed by the SPD, leaving very few samples per class. Thus, instead of discarding noisy samples, using them as unlabeled ones in SSL backbones is an effective way combat noisy labels.
The benefits of using SSL methods in our framework are in these aspects: (i) improving SS performance incrementally and (ii) leveraging the potentially noisy samples to learn feature representation. For (i), as shown Figure 2 (a) and (b), the precision of GPL is higher than DivideMir+ and DivideMir. This is be- cause the SSL backbone used in GPL, i.c., Pseudo- Labeling, can better alleviate the confirmation bias induced during the SS process and improve the prediction results. In this way, GPL can achieve competitive test accuracy in CIFAR-10 sym 80% without co-training two mod-
1.0
0.8
0.6
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
a) DivideMix+
b) GPL
Figure 3: Representations (t-SNE 2D embeddings) of two CIFAR- 10 classes, 'cat" and 'truck', learned by DivideMix+ (left) and GPL (right), with 80% noise rate. Blue/red dots represent clean samples of cat/truck, while magenta and cyan crosses represent corrupted samples.
11

---

Published in Transactions on Machine Learning Research (07 (07/2022)
els. This finding is more prominent in CIFAR-100 asym 40%. For (ii), we visualize feature representations of GPT, and Din-de.iir+ in 2-dimensional embeddings using i-SNE (der Maaten & Hinton 2008). Figure 3
presents the normalised 2D embeddings of 500 randomly selected samples from each of two classes in CIFAR- 10 sym 80%. We can see that both GPL and DivideMiz+ can accurately separate the clean instances of two classes (blue vs red). As for noisy instances (magenta and cyan), the representations learned by GPL are more fragmented and pushed away from clean clusters, showing that the SSL backbones of GPL is more effective in isolating noisy samples from clean ones. However, DivideMir+ outperforms GPL in test accuracy because DivideMiz+ leverage two models for training and testing to improve accuracy.
6.4 Efficiency analysis
In Table we study the efficiency of these two components by dividing all our instantiations into two parts and calculating their training time on CIFAR-10 sym 20% using an NVIDIA RTX 6000 GPU. We also break down the computation time per epoch (measured in seconds) for the SS process (Alg. 1, line 5) and SSL process (Alg. 1, line 6). For the efficiency of SS, DivideMix+ much faster than DivideMir because our per-minibatch SS strategy extracts and divides cach minibatch of data faster. We can conclude that instantiations from our framework work more efficiently than simply combining SS and SSL components. DivideMir- is faster than DivideMir, which shows that SPD (used by DivideMir-) is faster than GMM. Moreover, GPL is much faster because only one network is trained. For the efficiency of SSL, SPD-TE is faster than SPD-MM and SPD-PL because the last two methods need to take an additional Mizl'p operation. The small time differences between these three instantiations indicate that various SSL backbones can be integrated into our framework efficiently.
Table 7: Comparison of training time on CIFAR-10.
Method GPL
DivideMix+ DivideMix- DivideMix
SPD-CE
SPD-TE SPD-PL SPD-MM
Total
3.3 h
8.2 h
9.1 h
9.4 h
2.5 h
3.0 h
3.2 h
3.6 h
SS/epoch
9.2 s
12.1 s
15.5
18.2
37.2
39.5
44.1
6.5 The Effectiveness and Efficiency of Two Networks
To study the effectiveness of training two networks on our instantiations, we compare six variants of our framework on CIFAR-10: DivideMix (1 network), DivideMix (2 networks), DivideMix+ (1 network), DivideMix+ (2 notwork)), GPL (1 network), and GPL (2 networks). "1/2 network notwork/networkke nnttwokss"" mcans label co-refinement and co co-guessing (Li et al.) (2200)). The results of test accuracy and average training time are in Ta- ble g We observe that training with two networks provides a performance boost for the corresponding instantiations. The performance al the high noise rate is more significant than that of the medium noise rate. This is reasonable since the purpose of two diverged networks is avoid error accumulation (Li et al. 2020b) during the training, a high noise rate will induce more errors for the single model.
Table 8: Ablation Study of training with two networks
on different instantiations on CIFAR-10.
Model
Method Sym-40% Sym-80%
Asym 40%
Time
network
92.38
87.59
56.76
DivideMix
networks
93.76
89.33
90.66
network
92.50
90.66
89.36
5.4
DivideMix
the instantiation
2 networks
93.82
92.04
91.37
8.8
network
94.09
91.83
93.06
3.5
GPL
2 networks
94.68
93.38
93.75
5.2
7
Conclusion
This paper proposes versatile framework called SemiNLL for NLL. This framework consists of two main parts: the mini-batch SS strategy and the SSL backbone. We conduct extensive experiments on benchmark- simulated and real-world datasets to demonstrate that SemiNLL can absorb a variely of SS strategies and SSL backbones, leveraging their power to achieve state-of-the-ar performance in different noise scenarios. Morcover, we thoroughly analyze the effects of the two components in our framework. Recommendations on the choices of the components in our framework can be found in Appendix G In future work, we hope develop more advanced algorithms guided by this framework to tackle noisy labels.
12

---

Published in Transactions on Machine Learning Research (07 (022))
References
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In Proceedings of the International Conference on Machine Learning, 2019.
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Pscudo-labeling and confirmation bias in deep semi-supervised learning. In Proceedings of the International Joint Conference on Neural Nefworks, pp. 1-8, 2020.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, and Yoshua Bengio. A closer look at memorization in deep networks. In Proceedings of the International Conference on Machine Learning, 2017.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mix- match: holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 5049 5059, 2019.
Fengwen Chen, Guodong Long, Zonghan Wu, Tianyi Zhou, and Jing Jiang. Personalized federated learning with graph. arXiv preprint arXiv:2203.000 00829, 2022.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In Proceedings of the International Conference on Machine Learning. pp. 1062 1070, 2019.
Filipe Cordeiro, Ragav Sachdeva, Vasileios Belagiannis, lan Reid, and Gustavo Carneiro. Longremix: Robust learning with high confidence samples in noisy label environment. arXiv preprint arXiv:2103.04113: 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, and Fei-Fei Li. Imagenet: large-scale hierarchical image database. In Proceedings of the TEEE/CVE Conference on Computer Vision and Pattern Recognition, 2009.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(1), 2008.
Yifan Ding, Liqiang Wang, Deliang Fan, and Boqing Gong, A semi-supervised two-stage approach to learning from noisy labels. Tn Proceedings of the IEFE/CVF Winter Conference on Applications of Computer Vision, pp. 1215 1224, 2018.
Lei Feng, Senlin Shu, Zhuoyi Lin, Fenginao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In Proceedings of the International Joint Conferences on Artificial Intelligence, 2020.
Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning, pp. 1126- 2017.
Luis Garcia, Andre de Carvalho, and Ana Lorena. Noise detection in the meta-learning level. Newrocomput- ing, 176:14 25, 2016.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
Jacob Goldberger and Ehud Ben- Ren-Reuven. Training deep neural-networks using a noise adaptation layer. In Proceedings of the International Conference on Learning Representations, 2016.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Tvor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, 2018.
Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In Proceedings of the International Conference on Computer Vision, pp. 5138-57 5147, 2019.
13

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEF/CV Conference on Computer Vision and Patlern Recognilion, pp. 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in Neural Information Processing Systems, pp. 10456-10463, 2018
Peng Hu, Xi Peng, Hongyuan Zhu, Liagli Zhen, and Jie Lin. Learning cross-modal retrieval with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5403-5413, 2021.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5070-5079, 2019.
Jing Jiang, Shaoxiong Ji, and Guodong Long. Decentralized knowledge acquisition for mobile internet applications. World Wide Web, 23(5):2653 2669, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Fei-Fei Li. Mentornet: Learning data-driven cur- riculum for very deep neural networks on corrupted labels. In Proceedings of the International Conference on Machine Learning, pp. 2304-2313, 2018.
Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distribu- tion aligning refinery of pseudo-label for imbalanced supersuper learning. In Advances in Newral Information Processing Systems, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Blya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Abhishek Kumar, Avishek Saha, and Hal Daume. Co-regularization based semi-supervised domain adapta- tion. In Advonces in Neural Information Processing Systems, pp. 478 06-4466 2010.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Proceedings of the International Conference on Learning Representations, 2016.
Yann Lecun, Loon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, 86(11), 1998.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of the Workshop on International Conference on Machine Learning, 2013.
Kuang- Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5447 5456, 2018.
Bingcong Li, Bo Han, Zhuowei Wang, Jing Jiang, and Guodong Long. Confusable learning for large-class few-shot classification. In Joint European Conference on Machine Leurning and Knowledge Discovery in Databases, pp. 707-723. Springer, 2020a.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan Kankanhalli. Learning to learn from noisy labeled data. Tn Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5051-5059, 2019.
Junnan Li, Richard Socher, and Steven Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In Proceedings of the International Conference on Learning Representations, 2020b.
14

---

Published in Transactions on Machine Learning Research (07/2022)
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint XrXi:: 02862, 2017.
Yixin Liu, Shirui Pan, Yu Guang Wang, Fei Xiong, Liang Wang, Qingfeng Chen, and Vincent CS Lee. Anomaly detection in dynamic graphs via transformer. IEEE Trunsactions on Knowledge and Data En- gineering, 2021.
Yixin Liu, Yu Zheng, Daokun Zhang, Hongxu Chen, Hao Peng, and Shirui Pan. Towards unsupervised deep graph structure learning. In Proceedings of the ACM Web Conference 2022, pp. 1392-1422000 1403, 2022.
Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated learning for open banking. In Federated learning, pp. 240-2522-022 254. Springer, 2020.
Guodong Long, Tao Shen, Yue Tan, Leah Gerrard, Allison Clarke, and Jing Jiang. Federated learning for privacy-preserving open innovation future on digital health. In Humanity Driven AI, pp. 113-133. Springer, 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmen- tation. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognilion, pp. 3431-34 3440, 2015.
Tya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXio preprint arXiv:1608.0390 2016.
Jie Ma, Guodong Long, Tianyi Zhou, Jing Jiang, and Chengqi Zhang. On the convergence of clustered federated learning. aixiv preprint arXiv:2202. 201177202006187 2022.
Xingjun Ma, Yisen Wang, Michael Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionallty-drivee learning with noisy labels. In Proceedings of the International Conference on Machine Learning, pp. 3355-3364, 2018.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In Proceedings of the International Conference on Machine Learning, 2020.
Eran Malach and Shai Shalev-Shwarzz. Decoupling' when to update' from' how update'. In Advances in Newral Information Processing Systems, pp. 960-970, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicttion-cffciine learning of deep networks from docentralized data. In AISTATS, pp. 1273-1282. PMLR, 2017.
Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class-probability estimation. In Proceedings of the International Conference on Machine Learning, pp. 125 134, 2015.
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 19(6:12336 1246, 2018.
Nagarajan Natarajan, Inderjit Dhillon, Pradcep Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in Neural Information Processing Systems, pp. 1196 1204, 2013.
Due Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-ensembling. In Proceedings of the International Conference on Learning Representations, 2020.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
15

---

Published in Transactions on Machine Learning Research (07/2022)
Haim Permuter, Joseph Francos, and Ian Jermyn. A study of gaussian mixture models of color and texture features for image classification and segmentation. Paltern Recognition, 39(4):695-70) 706, 2006.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In Proceedings of the International Conference on Machine Learning. 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pp. 91-99, 2015.
Florian Schroff, Antonio Criminisi, and Andrew Zisserman. Harvesting image databases from the web. IEEE Transactions on Pattern Analysis and Mochine Intelligence, 33(4):754-766 2010.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. weigtweight- Learning an explicit mapping for sample weighting, In Advances in Neural Information Processing Systems, pp. 1919-1930, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Newral Information Processing Systems, pp. 4077-4087, 2017.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. In Advances in Neural Information Processing Systems, volume 33, 2020.
Hwanjun Song, Minseok Kim, and Jae- Lee. Selfie: Refurbishing unclean samples for robust deep learning, In Proceedings of the International Conference on Machine Learning, pp. 5907-5915, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929 1958, 2014.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang, Fedproto: Federated prototype learning over heterogeneous devices. arXiv preprint arXt::2055.0024 2021.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5552-5560, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, pp. 1195-1204 1204, 2017.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the International Conference on Computer Vision, pp. 322 330, 2019.
Zhen Wang, Guosheng Hu, and Qinghua Hu. Training noisc-robust deep neural networks via meta-learning, Proceedings of the (EEE/CVI Conference on Computer Vision and Pallern Recognition, pp. 4524 4533, 2020.
Zhuowei Wang, Tianyi Zhou, Guodong Long, Bo Han, and Jing Jiang. Fednoil: A simple two-level sampling method for federated learning with noisy labels. arXiv preprint arXin:2205.10 10110, 2022.
Chen Wei, Kihyuk Sohn, Clayton Mcllina, Alan Yuille, and Fan Yang. Crest: A class-rebalancing self- training framework for imbalanced supervupe learning. In Proceedings of the IEEE/CVF conference on computer vision and poltern recognition, pp. 10857-10866, 2021a.
16

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularivation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13726 13735, 2020.
Yi Wei, Xue Mei, Xin Liu, and Pengxiang Xu. Dst: Data selection and joint training for learning with noisy labels. arxiv preprint or 0rx::21003.005 2021b.
Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classification with instance- dependent label noise. In Pirst Conference on Causal Learning and Reasoning, 2021a.
Yichen Wu, Jun Shu, Qi Xic, Qian Zhao, and Deyu Meng. Learning to purify noisy labels via meta soft label corrector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021b.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In Proceedings of the International Conference on Learning Representations, 2022.
Han Xiao, Kashil Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv:1708.071 2017.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang- Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2691-2699, 2015.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, volume 33, pp. 6256-62666 6268, 2020.
Jingyi Xu, Zihan Chen, Tony QS Quek, and Kai Fong Ernest Chong, Fedcorr: Multi-stage federated learning for label noise correction. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 10184 10193, 2022.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L. dmi: An information--hoorriic noisc-robust. loss function. In Advances in Newral Information Processing Systems, 2019.
Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto policy pool for model-basod offline reinforcement learning. In International Conference on Learning Representations, 2021.
Yijun Yang, Jing Jiang, Zhuowei Wang, Qiqi Duan, and Yuhui Shi. Bies: Adaptive policy optimization for model-based offline reinforcement learning. In Australasion Joint Conference on Artificial Intelligence, pp. 570-581. Springer, 2022.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin-Yau Kwok. Searching to exploit mem- orization effect in learning with noisy labels. In International Conference on Machine Learning, pp. 10789 10798, 2020a.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. In Advances in Neural Information Processing Systems, volume 33, 2020b.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7017-7025, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 14129-14142, 2020.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagrocment help generalization against label corruption? In Proceedings of the International Conference on Machine Learning, 2019.
17

---

Published in Transactions on Machine Learning Research (07/2022)
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In Proceedings of the European Conference on Computer Vision, pp. 68 83, 2018.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems, volume 34, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In Proceedings of the International Conference on Learning Represenlations, 2016.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entr loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, pp. 8778 8788, 2018.
Zizhao Zhang, Han Zhang, Sercan Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noisc. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9294 9303, 2020.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: From clean label detection to noisy label self-correction. Proceedings of the International Conference on Learning Representations, 2021.
18

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Appendix
We provide a summary of contents below for easier navigation of the appendix of our paper titled "SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning".
Contents
Appendix
A
Datasets
Appendix
B
Network Structure
Appendix
C
Construction of Noisy Labels
Appendix
D
Additional Experiment Results
Appendix D.1
Results on More Real-world Datasets
Appendix D.2
Sensitivity to the Batch Size of SS
Appendix D.3
Results of Different Model Architectures
Appendix D.4
Comparison between GPL (mini-batch) and GPL (epoch)
Appendix
Limitations
Appendix
Broader Impact
Appendix
G
Choices of the Components in Our Framework
Appendix G.J
Choices of SS Scope
Appendix G.2
Choices of Number of Networks
Appendix G.3
Choices Different SS and SSL Combinations
A Appendix
Mustrations
19

---

Published in Transactions on Machine Learning Research (07 (2022)
A Datasets
MNIST and FASHION-MNIST contain 60K training images and 10K test images of size 28 x 28. CIFAR- 10 and CIFAR-100 contain 50K training images and 10K test images of size 32 x 32 with three channels. Clothing IM is a large-scale real-world dataset that consists of one million training images of size 224 x 224 from online shopping websites with labels annotated from surrounding texts. The estimated noise ratio is approximately 40% Xiao et al. (2015). Webvision Li et al. (2017) contains 2.4 million images crawled from the websites using the 1,000 concepts in ImageNet ILSVRCI Renng et al. (2009). Since the dataset is quite large, for quick experiments, we the first 50 classes of the Google image subset. We test the trained models on the human-annotated WebVision validation set. The detailed characteristies of the datasets in the experiments are shown in Table
Table 9: Summary of datasets used in the experiments.
of training
# of test
# of class
size
MNIST
60,000
10,000
10
1 x 28 x 28
F-MNIST
60,000
10,000
10
x 28 28
CIFAR-10
50,000
10,000
10
3 x 32 x 32
CIFAR-100
50,000
10,000
100
3 x 32 x 32
ClothingIM
1,000,000
10,000
14
x 224 x 224
B Network Structure
For MNIST, we use simple 2-layer MLP following Jocor Wei et al. (2020). For FASHION-MNIST, we use
ResNet-18 18 He et al. (2016) following GCE Zhang & Sabuncu (2018). For CIFAR-10 and CIFAR-100, we
use the "13-CNN" archilecture Tarvainen & Valpola (2017), which is shown in Table 10 For Clothing
we use ResNet-50 He et al. (2016) following DivideMia Li et al. (2020b). For mini Web Vision, we use the inception-resnetv2 Szegedy et al. (017).
Table 10: The "13-CNN" network architecture used in CIFAR-10 and CIFAR-100.
Layer Input
Hyperparameters
32 x 32 RGB image
Convolutional
128 filters, 3 x 3, some padding
Convolutional
128 filters, 3 x 3, same padding
Convolutional
128 filters, 3 x 3, same padding
Pooling
Maxpool 2 x
Convolutional
256 filters, x 3, same padding
Convolutional
256 filters, x 3, some padding
Convolutional
256 filters, x 3, same padding
Pooling
Maxpool 2 x 2
Convolutional
512 filters, 3 x 3, valid padding
Convolutional
256 filters, x 1, same padding
Convolutional
128 filters, x 1, some padding
Pooling
Average pool (6 x > 1x1 pixcls) Fully connected 128 10 (100)
Softmax
C
Construction of Noisy Labels
The datasets used in our experiments are FASHION-MNIST. CIFAR-10, and CIFAR-100. To add noisy labels, corrupt these datasets by two widely-used types of noisy labels: symmetric noise and asymmetric
20

---

Published in Transactions on Machine Learning Research (07 (022))
Following these works, mamually corrupt the dataset by the label transition matrix Q. where Qi p = jly i) given that noisy is flipped from clean For
noise Li et al. 2020b Zhou et al. 2021
symmetric noise, we inject the symmetric label noise as follows:
1-6
C-1
Q =
(5)
where C is the number of classes, and is the noise ratio. Asymmetric noise is well-known simulation of fine-grained classification with noisy labels, where annotators may make mistakes only within very similar
classes. Its noise transition matrix Q (laking 6 classes example) is obtained as follows:
0
0
1
0
6
1
0
0
(6)
0
0
0
1
0
0
0
0
0
1
0
0
6
0
0
1-
Additional Experiment Results
D.1 Results on More Real-world Datasets
We test two real-world datasets, Food-101N (Lee et al. 2018) and ANIMAT-11NN (Song al. 2019) Food-101N is dataset for food classification. It consists of 310,009 training images and 25,000 testing images in 101 classes collected from the web. The estimated label purity is 80%. We use ResNet-50 pre-trained on ImageNet and there are 30 epochs in total. ANIMAL-1ON contains 10 animals with confusing appearances downloaded online. There are 50,000 training and 5,000 testing images. The estimated label noise rate is 8%. We use VGG-19 with batch normalization and there are 100 epochs total. As shown Table [1 and Table [2 we can conclude that our instantiation, DivideMix+, consistently outperforms all base methods, including its epoch-wise variant, DivideMix. GPL achieves competitive results using single model.
Table 11: Test accuracy (%) on Food-101N.
Table 12: Test accuracy (%) on ANTMAT.1ION.
Method
Accuracy
Method
Accuracy
Gross-Entrowd
81.53
Cross-Entropy Entropy
79.4
NeeanNet (Lee et al.
2018
83.95
Dropout Srivastava et al.
2014
81.3
DeepSelf (Han et al.
2019
85.10
SELFIE (Sng et al. 2019
81.8
DivideMix (Li et al.
2020b
85.64
DivideMix
Li et al. 2020b
82.4
DivideMix+ (ours) GPL (ours)
86.93
DivideMix+ (ours) GPL (ours)
83.6
83.2
86.59
D.2 Sensitivity to the Batch Size SS
We conduct a sensitivity analysis on the batch size of our two instantiations, DivideMix+ and GPL, on CIFAR-10 Sym 80%. In Table 13 we observe that larger the batch size achieves better test accuracy since extremely high noise mini-batch can be avoided. Too large batch size might result in performance degradation because the model might be trapped in local optimum [Loshchilov & Hutter 2016
21

---

Published in Transactions on Machine Learning Research (07 (0222)
Table 13: The test accuracy (%) of DivideMix+ and GPL. with different batch sizes on CIFAR-10 Sym 80%.
Batch size
32
64
128
256
512
DivideMix+
90.36
91.41
91.97
92.04
91.63
GPL
88.46
90.16
91.81
91.83
91.03
D.3 Results of Different Model Architectures
We compared DivideMix, DivideMix+, and GPL on CIFAR-10 Sym-40%, Sym-80%, and Asym-40% noise with three different model architectures. We observe that DivideMix+ and GPL outperform DivideMix regardless of using deeper/wider networks. DivideMix- outperforms GPL under high symmetric noise, while GPL achieves better results on low noise ratio and asymmetric noisc.
Table 14: Comparison between DivideMix, DivideMix+, and GPL using different model architectures in test accuracy (%) on CIFAR-10. Key: WRN (Wide ResNet), PRN (PreActivation ResNet).
Model
Method
Sym-40%
Sym-80%
Asym-40%
DivideMix
93.76
89.33
90.66
13-CNN
DivideMix+
93.82
92.04
91.37
GPL
94.09
91.83
93.06
DivideMix
93.02
90.94
91.60
PRN-18
DivideMix+ GPL.
93.18
92.92
93.12
94.64
90.92
93.44
DivideMix
92.72
90.11
90.79
WRN-28
Dix
93.26
92.57
92.06
GPL.
93.35
90.52
92.28
D.4 Comparison between GPL (mini-batch) and GPL (epoch)
To demonstrate the effectiveness of our mini-batch SS, we also compare GPL (mini-batch) with GPL (epoch) with batch size of 256 on CTFAR-10 in Table 15 We observe that the test accuracy of GPL (mini-batch) constantly outperforms GPL (epoch).
Table 15: The test accuracy (%) of GPL. (epoch) and GPL. (mini-batch) CIFAR-10.
Method
Sym-40% Sym-80%
Asym-40%
GPL (cpoch) 93.13
88.96
91.55
GPL (mini-batehl
94.09
91.83
93.06
E Limitations
Mini-batch SS. Noise ratios in different mini-batches inevitably fluctuate since each mini-batch of data is randomly sampled from the whole datasct. When the overall noisc level is high and the batch size is too small, the SS strategy might have trouble distinguishing clean and noisy samples in some severely corrupted mini-batches, which may deteriorate the overall performance. For example, in DivideMixt, we apply GMM cach mini-batch of data. The fluctuation of the noisy ratio each mini-batch might result in inconsistent data selection criteria (clean threshold). Morcover, we conduct sensitivity analysis on the batch size of our two instantiations, DixideMix+ and GPL, on CIFAR-10 Sym 80% in Appendix D.2
22

---

Published in Transactions on Machine Learning Research (07 (07/2022)
SSL component. Under the symmetric noise, the labels are uniformly corrupted. If the SS strategy can distinguish clean/noisy samples well enough, the class distribution in the labeled/ mbbeledunlabelee unlabeled sets is uniform. While under the asymmetric noise, the labels are flipped to similar classes, resulting in the class- imbalanced labeled/ labeled/unlabeler unlabcled data. However, traditional SSL mothods (Berthelot et al., 2019 Arazo et al. 2020 Sohn et al., 2020) (including the SSL backbones of our instantiations) assume that the class distribution of labeled and/or or unlabeled data balancod. In Table (symmetric noise), methods using SSL (DivideMix, DivideMix+, and GPL) outperform others in most cases. In Table 2 aasymmetrie noise), methods using SSL perform slightly worse than other SOTA methods in MNIST and FASHION MNIST. Please be noted that our framework can absorb the advantages of various SS and SST, methods. To deal with the class- imbalanced issue in asymmetric noise, we can develop new instantiations using SSL methods designed to lackle class-imbalance data (Kim el al. 2020: Wei el. al. 2021a).
F Broader Impact
If the clean examples are incorrectly identified, the network could inadvertently ignore meaningful labels, and vice versa, i.e, the network could learn from meaningless noise rather than clean labels. This motivates a broad discussion about potentially catastrophic effects. We formalize a versatile framework that leverages SS to select clean samples and SSL to fully use the noisy samples by removing their labels. And we believe it will substantially impact labor-intensive jobs of checking data label quality, such as training models from the web-crawled images (Schroff et al., 2010) and medical data analysis (Miotto et al., 2018). One potential risk caused by the wrong sample selection process is the increased chances of the model over-fitting potential outliers in the data that may lead to erroncous or misleading results, i.c., the misdiagnosing of patients. In the future, we will develop more precious and safer SS strategies to alleviate this negative impact.
G Choices of the Components in Our Framework
The components considered in framework are: SS E {GMM, SPD), SSL E {Temppoal Ensembling, MixMatch, Psoudo-Labeling), SS scope E (Mini-batch, Epoch}, number of networks (1, 2}. We conduct more systematic comparison by considering all the components mentioned above. We also make some recommendations based on the comparison.
G.1 Choices of SS Scope
Analysis. The ad advantages and limitations of mini-batch SS compared to epoch-wise SS are thoroughly discussed in Section 3.1 and Appendix respectively.
SS over epoch- wise SS by comparing DivideMix and DivideMixt. In Appendix D.4 we also compare GPL (mini-batch) with GPL (epoch) on CIFAR-10. A sensitivity analysis of the batch size of mini-batch SS conducted in Appendix D.2
Empirical evidence. In Section 6.2 we demonstrated the benefit of
Our recommendation. If we choose bigger batch size, mini- batch-wist SS can be effective for both symmetric and asymmetric noisc. From the scnsitivity analysis on the batch size, the batch size needs to be bigger than 64 to avoid the fluctuation of the noisy ratio in each mini-batch. Thus, in the following comparisons, we only use mini- maii-batch-ise SS with a batch size 256.
G.2 Choices of Number of Networks
Analysis. The purpose of two diverged networks is to avoid error accumulation (Li et al. 2020bb during the training. Under high noise rates, more errors will be induced during the training of a single e model. Thus, training with two diverged networks is more effective in severely corrupted datasets. The drawback is the increase in training time and computational resources.
Empirical evidence. In Section 6.5 we demonstrated the effectiveness and efficiency of training with two networks.
23

---

Published in Transactions on Machine Learning Research (07/2022)
Our recommendation. Using two networks boost performance for the corresponding instantiations, especially at high noise rates. The shortcoming of using two networks is the increase in training time. So it might not be necessary at low noise rates or on simple datasets (MNIST). In the following comparisons on CIFAR-10, we will use two networks for all instantiations for a fair comparison.
G.3 Choices of Different SS and SSL Combinations
The methods considered in this work SS E {GMM, SPD), SSL (Temporal Ensembling, MixMatch, Pseudo-Labeling) We combine cach SS and each SSL method to conduct a systematic comparison in Table I6 We denole these instantiations as "SS" + "SSL". Please be noted that mini-batch SS and two networks are used in all the instantiations according to previous recommendations.
Table 16:
The Lest accuracy (%) instantiations of different SS and SSL methods under high/low symmetric
noise in CIFAR-10.
Noise type
SSYSSL
Temporal Ensembling
MixMatch
Pseudo Pseudo-Labeling Labeling
GMM
85. 79
93.82
94.68
Sym-40%
SPD
84.60
92.68
94.33
GMM
54.37
92.04
93.38
Sym-80%
SPD
51.56
90.98
92.20
GMM
64.53
91.37
93.75
Asym-40%
SPD
68. 74
92.11
92. 14
We discovered some general rules and synergistic pairs when combining different SS and SSL methods using our framework:
(i) General rules regarding SS: Under symmetric noise, instantiations with GMM achieve higher test accuracy than instantiations with SPD, while the results are reversed under asymmetric noise. This is because GMM distinguishes clean and noisy samples based on their loss distribution. For asymmetric noise, most samples have near-zero normalized loss due the low entropy predictions of the network (Li et al. 2020b), causing performance deterioration for GMM. Since SPD leverages the prediction of its network to choose clean samples, its performance is more sensitive to noisy rates rather than noise type.
Our recommendation. For symmetric noise and high noise ratios, GMM should be used as the SS stratogy. For asymmetric noise, SPD should be used as the SS stratogy. For low noise ratios, SPD might be more computationally efficient (Table 7 in achieving competitive test accuracy.
(ii) General rules regarding SSL: Instantiations with Temporal Ensembling are much worse than those with MixMatch and Pscudo-Labeling because Temporal Ensembling only uses consistency regularization for unsupervised loss. In general, instantiations with Pseudo-Labeling achieve higher accuracy than the other two because the regularization term used in Pseudo-Tabeling prevents the model from assigning all labels to a single class at the early training stage. From the results of Asym 40%, we observe performance degrade of all SSL methods, especially Temporal Ensembling, combined with the same SS method (even though SPD can alleviate the harm of asymmetric noise). This is because the class distribution in the labeled labeled/unlabelee belee sets after the SS process fass-imbalanced under the asymmetric noise. However, all the methods considered in this paper assume that the class distribution of labeled and/or unlabeled data is balanced.
Our recommendation. Our guidance is to select suitable SSL method for the corresponding la- /aled/unlabeled sets after the SS process instcad of "using the strongest SSL algorithm" to achieve the best. lest. accuracy. For example, the class distribution of labeled/ labeled/unlaber uulabeled sets is (close Lo) balanced, we can choose more advanced mainstream SSL methods (Sohn et al. 2020: Zhang et al. 2021] as long as we take into account computational resources and efficiency. If labelod sabeled/unlabelee /unlabeled sets are class- inaslanbalaneed, we
can choose SSL methods specifically designed tackle class-imbalance data (Kim al.) 2020 Wei et al. 2021a)
24

---

Published in Transactions on Machine Learning Research (07 (07/2022)
(iii) Synergistic combination: We surprisingly find that GMM Pseudo-Labeling, the two-network version of the original GPT,, achieves the best performance in all settings, including the asymmetric noise where SPD is supposed to be stronger.
Our recommendation. GMM Pscudo-Labeling achieves the best performance for both symmetric and asymmetric noise. We believe its superior performance benefits from the conceptually simple idea of Pseudo-Labeling reduce confirmation bias generated from the SS process. Please be noted that Pseudo- Labeling was not published at top conferences. That being said, we will keep exploring the possibility of more combinations of SS and SSL based on our framework to figure out the chemistry in between, instead of just simply piling up SOTA SS and SSL methods published in top conferences.
H Illustrations
To better understand the differences between DivideMia-, DivideMir, and DivideMia+, their illustrations are shown in Figure (a), (b), and (c), respectively.
Mini-batch n-1
MixMatch
X
DivideMix-
Epoch
Self- prediction divider
Mini-batch
MixMatch
TIME
U
Mini-batch n+1
MixMatch
a) DivideMix-
Mini-batch n-1
MixMatch
X
Mini-batch
TIME
DivideMix
Epoch
GMM
MixMatch
Ut
Mini-batch n+1
MixMatch
b) DivideMix
Mini-batch n-1
Xm
TIME
Mini-batch n
DivideMix+ Epoch
GMM
MixMatch
Um
Mini-batch n+1
c) DivideMix+
Figure 4: Comparisons between: (a) DivideMirz (b) DivideMix, and (c) DivideMir+. Squares represent
data. Circles represent SS strategy. Octagons represent SSL backbone.
25


Reviews:
[{"review_id": "8HMqrvgy9S", "reviewer": "Reviewer_e8Ug", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The proposed SemiNLL framework is intuitive and easy to comprehend.\n* A breadth of datasets, noise levels, and baselines were used in the experimental comparisons.\n* Writing is good quality.\n\nWeaknesses\n* The experimental comparison among instantiations of SemiNLL is not comprehensive. Not all combinations of the components were tried, and so the takeaways of the experiments are somewhat unclear. The most clear is the benefit of mini-batch vs. epoch-level SS. Beyond this, the conclusions become more unclear. See e.g. Sec. 6.3 which makes the point that GPL is more effective in separating noisy samples, yet DivideMix+ has better performance due to it having two models. However, in Sec. 4.2 it was stated that training two networks is out of scope. Yet this choice leads prevents an apples to apples comparison from being made.\n* The message about how to choose specific SS and SSL algorithms is unclear. Currently it seems that the guidance is to use the strongest SS and the strongest SSL algorithm. Are certain pairings of SS and SSL algorithms more synergistic than others, i.e. the sum is greater than the parts? Addressing questions like this are important to prevent future research from \"reinventing the wheel\", which was stated as a goal of the work.\n* No discussion of limitations of the proposed method. One point in particular would be to discuss the effect of batch size on the mini-batch SS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework called SemiNLL for performing deep learning in the presence of noisy labels. SemiNLL has two main components: (1) a sample selection (SS) algorithm which predicts whether examples are clean or noisy, and (2) a semi-supervised learning (SSL) algorithm that infers new labels for the noisy examples. The key benefit of SemiNLL is its modularity since a variety of SS or SSL algorithms can be used in the framework. Two main instantiations of SemiNLL are investigated: DivideMix+ (GMM + MixMatch with mini-batch sample selection) and GPL (GMM + PseudoLabel). Experiments are conducted on a variety of image datasets with both symmetric and asymmetric noise and the proposed instantiations were shown to have SOTA performance.\n\nContributions include:\n* Formalization of a framework called SemiNLL for noisy label learning that combines SS and SSL.\n* Both the DivideMix+ and GPL instantiations are shown to have very strong accuracy for both symmetric and asymmetric noise.\n* Demonstration of the benefit of mini-batch SS vs. epoch-level SS (DivideMix+ vs. DivideMix).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Each of the following are important:\n* Add a more systematic comparison of instantiations of SemiNLL. Of the components considered in this work, from what I gather, SS = {GMM, SPD}, SSL = {MixMatch, PseudoLabel, Temporal Ensembling}, SS scope = {Mini-batch, Epoch}, # of networks = {1, 2}. Only a subset of these possibilities are considered. The chosen instantiations, in particular DivideMix+ and GPL, do indeed achieve good performance but the work would have greater impact if a more systematic comparison can be made. Of these, comparing 1 vs. 2 networks seems to be the most important since as mentioned above the comparison with respect to GPL is currently an apples to oranges comparison.\n* Add discussion that makes a recommendation, based on empirical evidence, for how future researchers should go about selecting SS and SSL algorithms within the SemiNLL framework.\n* Add discussion of limitations of the proposed approach.", "broader_impact_concerns": "Regarding the use of SS and SSL: if the clean examples are incorrectly identified then the network could inadvertently ignore labels that actually are meaningful, and vice versa, i.e. the network could learn from meaningless noise rather than clean labels. This could have catastrophic effects depending on the application. A discussion of these potential risks should be added.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OAG4z49bZB", "reviewer": "Reviewer_8Gpa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed pipeline is straightforward.\n\nWeaknesses:\n\nMajor problems with the paper are over-claims, unclear definitions and methods, and limited novelty:\n\n(1) Definition of noisy labels: In the paper, noisy labels are defined as unlabelled sets. For instance, it is stated that “Another recent method, DivideMix (Li et al., 2020), leverages Gaussian Mixture Model (GMM) (Permuter et al., 2006) to distinguish clean (labeled) and noisy (unlabeled) data, and then uses a strong SSL backbone called MixMatch (Berthelot et al., 2019).”\n\nHowever, noisy labels can be also defined by inducing noise to the labels, while the paper simply considers unlabelled data (according to the statements). Please define “a noisy label” more precisely, and update the related parts accordingly.\n\n(1.a) Following the above issue, it is not clear how “noisy samples” are selected or constructed in the experimental analyses.\n\n(1.b) The paper proposes 2 methods for SS. \n\nFirst, GMM based methods.\n\nSecond: “The self-prediction divider which determines potentially clean samples in a mini-batch if the samples’ maximal likelihood predictions of the network match their annotated labels. Specifically, the samples are divided into the labeled set only if the model predicts the annotated label to be the correct class with the highest likelihood.”\n\nHowever, how these GMM and max-likelihood estimation models are trained and employed are not explained. They are just defined as black-boxes which split data into clean labelled /noisy (unlabelled?) sets. \n\n(1.c) Novelty of SS: The main difference between the proposed SS and related work is mini-batch sample selection. It is not clear why this method should work compared to the other baseline. Indeed, in the analyses, this method performs on par in various setups.\n\n(2) SSL: The paper briefly ensembles different SSLs with different SSs, and experimentally analyze their accuracy. However, stronger experimental and theoretical analyses are required to justify the results. In addition, it is claimed that they are “naturally combined”, however, this claim is vague and should be precisely supported.\n\n(3) The paper claims that the proposed method \"outperform other state-of-the-art noisy-label learning methods.\" However, in the analyses, various different combinations of cascaded integration of SSs and SSLs are given, and the proposed methods perform on par with, or slightly worse/better compared to the state-of-the-art. To resolve this, first experimental results should be described more precisely revising the claim. Second, additional analyses should be performed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework called SemiNLL  that integrates SS and SSL to improve accuracy of deep learning models. The SemiNLL was examined on Mnist, Fashion Mnist, Cifar-10 and 100 datasets for image classification tasks. In the analyses, the proposed SemiNLL performs on par with, or slightly worse/better compared to the state-of-the-art.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Some minor issues: \n\nThe abstract should be clarified. There are two major parts to be improved. First, describe the main novelty; that is, in “naturally combine different SS and SSL components”, please briefly describe “natural combination” and components. Second, please provide main results, i.e. describe what “the new state-of-the-art” is.\n\nSome of the requested changes were provided in the previous section.\n\nBriefly:\n\n(1) Several parts and statements of the paper should be clarified and more precise definitions should be provided.\n\n(2) The claims should be revised and updated.\n\n(3) Additional theoretical and experimental results should be given to support the claims on the novelty, if the paper is considered as an analysis paper. If authors consider that the main contribution of the paper is algorithmic, then novelty of the proposed framework should be improved by a novel method integrating SS and SSL. For the moment, it is not clear how \"natural\" their combination is.\n\n(4) The superiority of the proposed method over state-of-the-art is not clear. Additional analyses with different larger scale datasets and models should be provided for justification.\n\n", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sXoBsF6M8R", "reviewer": "Reviewer_TbTQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors revisit noisy label learning based on their usage of the Sample Selection (SS) strategy and Semi-Supervised Learning (SSL). A new framework to combine these two important components is proposed. The main idea of the framework is simple, and the authors claim the importance of the mini-batch sample selection as well as the SSL backbones. Various combinations of SS and SSL are investigated in the paper. The authors also compare the proposed method with various methods on synthetic and real datasets.\n\nWeakness:\n1. The main idea of the paper is not clear after reading the introduction. \n2. The proposed SemiSLL alternatively applies SS and SSL to update the model, whose effectiveness is only supported by some empirical results. The importance of the change should be discussed in more detail.\n3. The authors compare SemiSLL with previous methods in Section 3, but the possible drawbacks of the method are not fully discussed. For example, the mini-batch sample selection can mitigate the confirmation bias, but will make the SS more difficult with limited batch-size, since some methods need a holistic evaluation of data to determine the clean and noisy data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes SemiNLL, a versatile framework for noisy label learning. It combines the advantages of sample selection strategy and semi-supervised learning in a better manner. The results show promising performance.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please refer to the weakness part.\n\nThe authors may analyze the framework in more detail and show the mini-batch-wise usage of SS and SSL is more important than the epoch-wise usage of SS and SSL. \n\nMore real-world datasets should be compared. \n\nThe proposed method may depend on the assumptions of SSL. On which types of noisy learning datasets the method works well should be discussed.", "broader_impact_concerns": "No Broader Impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes a semi-supervised learning framework with the presence of noisy labels, named SemiNLL. The proposed SemiNLL is sound, and the experimental results are shown sufficiently. 

The AE has three minor suggestions: 1) Please proofread the entire paper to fix all the grammatical errors and typos; 2) the definition of noisy labels and the description of sample selection methods should be placed into the main text instead of the appendix; 3) make all the notations consistent throughout the entire paper. 

Per the above comments, the AE therefore recommends “Accept with minor revision”.

Decision: Accept (poster)