DivideMix: Learning with Noisy Labels as Semi-supervised Learning
Authors: Junnan Li
Richard Socher
Steven C.H. Hoi
Venue:  (2020)
Keywords: label noise
semi-supervised learning

Abstract: Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Code is available at https://github.com/LiJunnan1992/DivideMix .

Published as a conference paper at ICLR 2020
DIVIDEMIX: LEARNING WITH NOISY LABELS AS SEMI- SEME-SUPERVISED LEARNING
Jumnan Li, Richard Socher, Steven C.H. Hoi
Salesforce Research {junnan. 1i, rsocher, ahoi/esalesforre.ccm
ABSTRACT
Deep neural networks are known be amotation-hungrry Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss dis- tribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi- semi-supereised manner. To avoid confirmation bias, simultaneously train works where each network uses the dataset division from the other network. Dur- ing the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-ar methods. Code is available at https://qithus com LiwunnanI 192/
diverged net-
INTRODUCTION
The remarkable success in training deep neural networks (DNNs) is largely attributed the collection of large datasets with human annotated labels. However, it is extremely expensive and time-consuming to label extensive data with high-quality annotations. On the other hand, there exist alternative and inexpensive methods for mining large-scale data with labels, such as querying commercial search engines (Li et al.) 2017a), downloading social media images with tags (Mahajan et al. 2018). leveraging generatted labels (Kuznetsova et al. 2018), or using single annotator to label each sample (Tanno et al.) 2019). These alternative methods inevitably yield samples with noisy labels. A recent study (Zhang al. (017) shows that DNNs can easily overfit to noisy labels and results in poor generalization performance.
Existing methods on learning with noisy labels (LNL) primarily take a loss correction approach. Some methods estimate the noise transition matrix and use it to correct the loss function (Patrini et al. 2017 Goldberger & Ben- Reuven 2017). However, correctly estimating the noise transition matrix is challenging. Some methods leverage the predictions from DNNs to correct labels and modify the loss accordingly (Reed et al. 2015 Tanaka et al., 2018). These methods do not perform well under high noise ratio as the predictions from DNNs would dominate training and cause overfitting. To overcome this, Arazo al. (2019) adopt MixUp (Zhang al., 2018) augmentation. Another approach selects or reweights samples so that noisy samples contribute less to the loss (Jiang et al. 2018 Ren et al.) 2018). A challenging issue is to design a reliable criteria to select clean samples. has been shown that DNNs tend to learn simple patterns first before fitting label noise (Arpit al. 2017). Therefore, many methods treat samples with small loss as clean ones (Jiang et al. 2018 Arazo et al. 2019). Among those methods, Co-teaching (Han et al. 2018) and Co-teaching Yu et al. (019) train two networks where each network selects small-loss losss-loss-ae samples in a batch to train the other.
Another active area of research that also aims to reduce annotation cost is semi-supervised learning (SSL). In SSL, the training data consists of unlabeled samples in addition to the labeled samples. Significant progress has been made in leveraging unlabeled samples by enforcing the model to produce

---

Published as a conference paper at ICLR 2020
low entropy predictions on unlabeled data (Grandvalet & Bengio 2004
consistent predictions
on perturbed input (Laine & Aila 2017 Tarvainen & Valpola| 2017 Miyato et al. 2019). Recently, Berthelot et al. (2019) propose MixMatch, which unifies several dominant SSL approaches in one framework and achieves state-of-ther tor---r-ttt--ttttt performance.
Despite the individual advances in LNL and SSL, their connection has been underexplored. In this work, we propose DivideMix, which addresses learning with label noise in a semi-supervised manner. Different from most existing LNL approaches, DivideMix discards the sample labels that are highly likely to be noisy, and leverages the noisy samples as unlabeled data to regularize the model from overfitting and improve generalization performance. The key contributions of this work are:
We propose co-divide, which trains two networks simultancously. For each network, we dynamically fit Gaussian Mixture Model (GMM) on its per-sample loss distribution to divide the training samples into a labeled set and an unlabeled set. The divided data is then used to train the other network. Co-divide keeps the two networks diverged, so that they can filter different types of error and avoid confirmation bias in self-training
During SSL. phase, we improve MixMatch with label co-refinement and co co-guessing to account for label noise. For labeled samples, we refine their ground-truth labels using the network's 's predictions guided by the GMM for the other network. For unlabeled samples, we use the ensemble of both networks to make reliable guesses for their labels.
We experimentally show that DivideMix significantly advances state-of-the-an art results on multiple benchmarks with different types and levels of label noise. We also provide extensive ablation study and qualitative results to examine the effect of different components.
RELATED WORK
2.1
LEARNING WITH NOISY LABELS
Most existing methods for training DNNs with noisy labels seek to correct the loss function. The correction can be categorized in two types. The first type treats all samples equally and correct loss either explicitly or implicitly through relabeling the noisy samples. For relabeling methods, the noisy samples are modeled with directed graphical models (Xiao et al. 2015). Conditional Random Fields (Vahdat 2017), knowledge graph (Li et al. (017b), or DNNs (Veit et al. 2017]] Lee et al. 2018). However, they require access to small set of clean samples. Recently, [Tanaka
(Tanakk et al. (2018) and Yi & Wu (2019) propose iterative methods which relabel samples using network predictions. For explicit loss correction. Reed et al. (2013) propose bootstrapping method which modifies the loss with model predictions, and Ma et al. (2018) 8) improve the bootstrapping method by exploiting the dimensionality of feature subspaces, Patrini et al. (2017) estimate the label corruption matrix for loss correction, and Hendrycks et al. (2018) improve the corruption matrix by using clean set of data. The second type of correction focuses on reweighting training samples or separating clean and noisy samples, which results in correcting the loss function (Thulasidasan et al. 2019: Konstantinov & Lampert 2019). common method is to consider samples with smaller loss as clean ones (Shen & Sanghavi 2019). Jiang et al. (2018) train mentor network guide student network by assigning weights to samples. |enn et al. (2018) reweight samples based on their gradient directions. [Chen et al.) (2019) apply cross validation identify clean samples. Arazo al. (2019) calculate sample weights by modeling per-sample loss with mixture model. Han et al. (2018) train two networks which select small-loss samples within each mini-batch to train each other, and Yu et al. (2019) improve by updating the network on disagreement data to keep the two networks diverged.
Contrary to all aforementioned methods, our method discards the labels that are highly likely to be noisy, and utilize the noisy samples as unlabeled data to regularize training in SSL manner. Ding al. (2088) and Kong et al. (2019) have shown that SSL method is effective in LNL.. However, their methods do not perform well under high levels of noise, whereas our method can better distinguish and utilize noisy samples. Besides leveraging SSL, our method also introduces other advantages. Compared to self-training methods (Jiang et al. 2018 Arazo et al.) 2019). our method can avoid the confirmation bias problem (Tarvainen & Valpola (017) by training two networks to filter error for each other. Compared to Co-teaching (Han et al. 2018) and Co-teaching+ (Yu et al. 2019), method is more robust to noise by enabling the two networks to teach each other implicitly at each epoch (co-divide) and explicitly each mini-batch (label co-refinement and co-guessing).
2

---

Published as a conference paper at ICLR 2020
Mini-batch 1
Mini-batch 2
A
GMM
x
A
MixMatch
A
MixMatch
B
GMM
X
MixMattch
MixMatch
Epoch e-1
Co-Divide
Epach
Figure DivideMix trains two networks (A and B) simultaneously. At each epoch, network models its per-sample loss distribution with a GMM to divide the dataset into a labeled set (mostly clean) and an unlabeled set (mostly noisy), which is then used as training data for the other network (i.e. co-divide). At each mini-batch, network performs semi-supervised training using an improved MixMatch method. We perform label co-refinement on the labeled samples and label co-guessing on the unlabeled samples.
2.2
SEMI-SUPERVISED LEARNING
SSL methods aim to improve the model's performance by leveraging unlabeled data. Current state-of-the-art SSL methods mostly involve adding an additional loss term on unlabeled data to regularize training. The regularization falls into two classes: consistency regularization (Laine & Aila 2017 Tarvainn & Valpola 2017: Miyato et al. 2019) enforces the model to produce consistent predictions augmented input data; entropy minimization (Grandvalet & Bengio, 2004 Lee (013) encourages the model to give nigh-confidence predictions on unlabeled data. Recently, Berthelot et al. (2119) propose MitMatch, which unifies consistency regularization, entropy minimization, and the MixUp P (Zhagg et al. 2018 regularization into one framework.
METHOD
In this section, we introduce DivideMix, our proposed method for learning with noisy labels. An overview of the method is shown in Figure To avoid confirmation bias of self-training where the model would accumulate its errors, we simultaneously train two networks to filter errors for each other through epoch-level implicit teaching and batch-level explicit teaching. At each epoch, perform co-divide, where one network divides the noisy training dataset into a clean labeled set (X) and noisy unlabeled set (If), which are then used by the other network. At each mini-batch, one network utilizes both labeled and unlabeled samples to perform semi-supervised learning guided by the other network. Algorithm delineates the full algorithm.
3.1 CO-DIVIDE BY LOSS MODELING
Deep networks tend to learn clean samples faster than noisy samples (Arpit et al. 2017). leading to lower loss for clean samples (Han al. 2018 Chen et al. 2019). Following Arazo et (2019), we aim to find the probability of a sample being clean by fitting mixture model to the per-sample loss distribution. Formally, let (X,3) = ((r., y)) denote the training data, where x; is an image and yı {0, 1J is the one-hot label over C' classes. Given model with parameters 8, the cross-entropy loss E(6) reflects how well the model fits the training samples:
C
(()) = (4) =
Σ of log (2; (zy; 0))
(1)
where Phode is the model's 's output softmax probability for class e.
Arazo et al. (2019) fit two-component Beta Mixture Model (BMM) to the max-normalized loss É to model the distribution of clean and noisy samples. However, we find that BMM tends to produce undesirable flat distributions and fails when the label noise is asymmetric. Instead, Gaussian Mixture Model (GMM) (Permuter et al. 2006) can better distinguish clean and noisy samples due to its flexibility in the sharpness of distribution. Therefore, the pnneectation-Mxximiaae probability p(g) where g is the Gaussian component with smaller mean (smaller loss).
fit two-component GMM & using algorithm. For each sample, its clean probability w; is the posterior
We divide the training data into a labeled set and an unlabeled set by setting a threshold T on W;- However, training model using the data divided by itself could lead to confirmation bias (i.e. the
3

---

Published as a conference paper at ICLR 2020
Algorithm I: DivideMix. Line 4-8: co-divide; Line 17-18: label co-refinement; Line 20: label co-guessing.
Input: g(3) and g(2), training dataset (X,) (2), clean probability threshold т. number of augmentations M, sharpening temperature T. unsupervised loss weight Au. Beta distribution parameter for MixMatch. I/ standard training (with confidence penalty)
00,,0 (1),0()) WarmUp(A'. (/0()) (00))(()e))))) .0(2))
while < MaxEpoch do
Wr2) = GMM(X, V (())
I/ model per per-sammle loss with ( to obtain clean proabability for g
W = GMM(X, V, )
I/ model per-sample loss with 8 to obtain clean proabability for g()
for h: 1,2 do
I/ train the two networks one by one
xx ((r yr, wi; ww)/u 2 -, V(r,, yr, wx) ) E (X. W(()
labeled training set for 00%)
(e = faulur < T, v(a,, wt) E (X, m(h))
Il unlabeled training set for ((%)
for iter = 1 to num_iters do
From X2 draw mini-batch {as, Ub, ws);; b E (1..... B)}
From 2 draw mini-hatch (us: E (ly...))))
12
for
do
13
for m = 1 to M do
fa.m A Augment(12)
Il apply whth round augmentation to xbb
im Augment(us)
Il apply m²th round of augmentation to us.
16
end
17
Pb M Em. Proddi
average the predictions across augmentations of me
bb whyb (1 wh)ps
refine ground-truti label guided by clean probability produced by the other network
19
0b = Sharpen(To. T)
I1 apply temperature sharpening to the refined label
46 अात Em (Us, 0(1) + Pmodel (us ()))
20
// co-guessing: average the predictions from both networks across augmentations of
Q6 Sharpen(q.. T)
11 apply temperature sharpening to the guessed label
22
end
23
= (@ 2b); be E (I, B). E (L)
Il augmented labeled mini-batch I| augmented unlabeled mini-batch Il apply MixMatch I1 total loss
24
ù [( 90): E (1..., B). E (1....,M)
25
Lx. Lu = MixMatch(((21)
C = Lx AnLCu Arfrng
g(k) SGD(E, 0(K))
Il update model parameters
28
end
end
30 end
model is prone to confirm its mistakes Tarvainen & Valpola, 2017)), as noisy samples that are wrongly grouped into the labeled set would keep having lower loss due the model overfitting to their labels. Therefore, we propose co-divide to avoid error accumulation. In co-divide, the GMM for one network is used divide training data for the other network. The networks kept diverged from each other due to different (random) parameter initialization, different training data division, different (random) mini-batch sequence, and different training targets. Being diverged offers the two networks distinct abilities to filter different types of error, making the model more robust to noise.
Confidence Penalty for Asymmetric Noise. For initial convergence of the algorithm, we need to "warm up" the model for a few epochs by training on all data using the standard cross-ent cross-entropy loss. The warm up is effective for symmetric (i.e. uniformly random) label noise. However, for asymmetric (i.e. class-conditional) label noise, the network would quickly overfit to noise during warm up and produce over-confident (low entropy) predictions, which leads to most samples having near-zero normalized loss (see Figure p) In such cases, the GMM cannot effectively distinguish clean and noisy samples based on the loss distribution. To address this issue, we penalize confident predictions from the network by adding negative entropy term, - (Pereyra al. 2017), 1,), to the cross-entropy loss during warm up. The entropy of a model's prediction for an input x is defined as:
H Σ Pinndee (enndel(1 (z; 0) log(Pinda 0)),
By maximizing the entropy, becomes more evenly distributed (see Figure (p) and easier to be modeled by the GMM. Furthermore, in Figure 2p we show & when the model is trained with
4

---

Published as a conference paper at ICLR 2020
Epoch 10: Cross Entropy
Confidence Penally
Epoch 20: DivideMii:
10
Noiey
Empliccll pel
Empircall
Empircall
0.0
0.2
0.6
0.8
1.
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.4 0.6
0.8
1.
Normalized loss 8
Normalized
Normalized loss
(a)
(b)
(c)
Figure 2: Training on CIFAR-I0 10 with 40% asymmetric noise, warm up for 10 epochs. (a) Standard training with cross-entropy loss causes the model to overfit and produce over-confident predictions, making & difficult to be modeled by the GMM. (b) Adding confidence penalty (negative entropy) during warm up leads more evenly-distributed f. (c) Training with DivideMix can effectively reduce the loss for clean samples while keeping the loss larger for most noisy samples.
DivideMix for 10 more epochs after warm up. The proposed method significantly reduce the loss for clean samples while keeping the loss larger for most noisy samples.
3.2 MIXMATCH WITH LABEL CO- CO-REFINEMENT AND CO-GUESSING
At each epoch, having divided the training data, we train the networks one time while keeping the other one fixed. Given a mini-batch of labeled samples with their corresponding one-hot labels and clean probability, {Ir, yb. wb); b E (1.... B)), and a mini-bated of unlabeled samples {ug: b (1, B)}. we exploit MixMatch (Berthelot et al.) 2019) for SSL. MixMatch utilizes unlabeled data by merging consistency regularization (i.e. encourage the model to output same predictions perturbed unlabeled data) and entropy minimization (i.e. encourage the model to output confident predictions on unlabeled data) with the MixUp (Zhang et al. 2018) augmentation (i.e. encourage the model to have linear behaviour between samples).
To account for label noise, we make two improvements to MixMatch which enable the two networks to teach each other. First, perform label co refinemeent for labeled samples by linearly combining the ground -ruth label y/ with the network's 's prediction ps (averaged across multiple augmentations of st)). guided by the clean probability wi produced by the other network:
f& waYb (1 Wb)Pb.
(3)
Then apply a sharpening function the refined label to reduce its temperature:
j Sharpen(Je. T) = 38 +
Σ गズ for c 1,2,... C.
(4)
Second, the ensemble of predictions from both networks "co-guess" the labels for unlabeled samples (algorithm) | line 20), which can produce more reliable guessed labels.
Having acquired (and L7) which consists of multiple augmentations of labeled (unlabeled) samples and their refined (guessed) labels, we follow Mixhatch to "mix" the data, where each sample is interpolated with another sample randomly chosen from the combined mini-batch of & and ú. Specifically, for a pair of samples (x1, 22) and their corresponding labels (Pı,P2), the mixed (x', p²) computed by:
À Beta(a, a),
(5)
X = max A,, 1,1 ->))
(6)
Nxi + (1 - 22)
(7)
p' Xpi X'222
5

---

Published as a conference paper at ICLR 2020
MixMatch transforms & and Li into X' and U' Equation 6 ensures that 2" are "closer" to & than U. The loss X' the cross-entropy loss and the loss on 2 the mean squared error:
Lx
Σ x.PEX"
Σ
Pe (2;
(9)
X
Lu
Σ Z.PER''
lp
(m; (1;
(10)
Under high levels of noise, the network would be encouraged to predict the same class to minimize the loss. To prevent assigning all samples to single class, we apply the regularization term used by [Tanaka et al. (2018) and Arazo et al. (2019), which uses uniform prior distribution (i.e. m. 1/C) regularize the model's average output across all samples in the mini-batch:
Lreg Σ ne log Te
Pmmodel (x; 0)
(11)
+ [
xEX'42
Finally, the total loss is:
C == + XucLer + X Lreg.
(12)
In our experiments, we set A, as and use Xu to control the strength of the unsupervised loss.
EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
We extensively validate method four benchmark datasets, namely CIFAR- 10, CIFAR
100 (Krizhevsky Hinton 2009). Clothing 1M (Xiao et al. 2015), and Web vision (Li et al. 2017a). Both CIFAR-10 and CIFAR-T00 contain 50K training images and 10K test images of size 32 x 32. Following previous works TTnaka et al.) 2018 Li et al. 2019). we experiment with two types of label noise: symmetric and asymmetric. Symmetric noise is generated by randomly replacing the labels for a percentage of the training data with all possible labels. Note that there is another criterion for symmetric label noise injection where the true labels cannot be maintained (Jiang al., 2018 Wang et al. 2018), for which we also report the results (Table Appendix). Asymmetric noise is designed to mimic the structure classes (e.g. hhoree dog+>>cat).
label noise, where labels are only replaced by similar
We use an 18-layer PreAct Resnet (He et al. 2016) and train it using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128. The network is trained for 300 epochs. We set the initial learning rate as 0.02, and reduce it by a factor of 10 after 150 epochs. The warm up period is 10 epochs for CIFAR-10 10 and 30 epochs for CIFAR-100 100. We find that most hyperparameters introduced by DivideMix do not need to be heavily tuned. For all CIFAR experiments, we use the same hyperparameters M 2, T 0.5, and 4. is set as 0.5 except for 90% noise ratio when is set as 0.6. We choose A, from {0, 25, 50, 150} using a small validation set.
Clothing IM and WebVision 1.0 are two large-scale datasets with real-world noisy labels. Clothing consists of million training images collected from online shopping websites with labels generated from surrounding texts. We follow previous work (Li et al. 2019) and use ResNet-50 with ImageNet pretrained weights. WebVision contains 2.4 million images crawled from the web using the 1,000 concepts ImageNet ILSVRCI2. Following previous work (Chen al. 2019), we compare baseline methods on the first 50 classes of the Google image subset using the inception-resner v2 (Szegedy al. 2017). The training details are delineated in Appendix
4.2
COMPARISON WITH STATE-OF-THE-WT OF-THE-ART METHODS
We compare DivideMix with multiple baselines using the same network architecture. Here we introduce some of the most recent state-of-the- art methods: Meta-Learning (Li et al., 2019) proposes gradient based method to find model parameters that
noise-tolerant; Joint- Optim (Tanaka et al., 2018) and P-correction Yi & Wu 2019) jointly optimize the sample labels and the network parameters; M-correction Arazo et al.
2019) models sample loss with BMM and applies MixUp.

---

Published as a conference paper at ICLR 2020
Dataset
CIFAR-I0
CIFAR-100
Method//Noise ratio
20%
50%
80%
90%
20%
50%
80%
90%
Best
86.8
79,4
62.9
42.7
62.0
46.7
19.9
10.1
Cross-Ener Coos-Entropy
Last
82.7
57.9
26.1
16.8
61.8
37.3
8.8
3.5
Bootstrap
Best
86.8
79.8
63.3
42.9
62.1
46.6
19.9
10.2
Reed
2015
Last
82.9
58.4
26.8
17.0
62.0
37.9
8.9
3.8
F-correctinn Patrini et al.
Best
86.8
79.8
63.3
42.9
61.5
46.6
19.9
10.2
2017
Last
59.4
26.2
18.8
61.4
37.3
Co-teaching
Best
89.5
85.7
67.4
47.9
65.6
51.8
27.9
13.7
Yu 2019
Last
88.2
84.1
45.5
30.1
64.1
45.3
15.5
8.8
Mixup
Best
95.6
87.1
71.6
$2.2
67.8
57.3
30.8
14.6
Zhang et al
2018
Last
92.3
77.6
46.7
43.9
46.6
17.6
8.1
P-correction'
Best
92.4
89.1
77.5
58.9
69.4 .
57.5
31.1
15.3
Yi Wu
2019
Last
92.0
88.7
76.5
58.2
68.1
56.4
20.7
Meta-Leaming Li al. 2019)
Best
92.9
89.3
77.4
58.7
68.5
59.2
42.4
19.5
Last
92.0
88.8
76.1
58.3
67.7
58.0
40.1
14.3
M-correction
Best Last
92.0
86.8
69.1
73.9
66.1
48.2
24.3
Arazo et 2019
93.8
91.9
86.6
68.7
73.4
65.4
47.6
20.5
Best
96.1
94.6
93.2
74.6
60.2
31.5
DivideMix
Last
95.7
94.4
92.9
75.4
76.9
74.2
59.6
31.0
Table I: Comparison with state-of-the-art methods in test accuracy (%) on CIFAR-I0 and CIFAR-100 with symmetric noise. Methods marked by denote re-implementations based on public code.
Note that none of these methods can consistently outperform others across different datasets. M- correction excels at symmetric noise, whereas Meta-Learning performs better for asymmetric noise.
T shows the results on CIFAR-10 and CIFAR-100 with different levels of symmetric label noise ranging from 20% to 90%. We report both the best test accuracy across epochs and the averaged test accuracy over the last 10 epochs. DivideMix outperforms state-of-the-arr methods by large margin across all noise ratios. The improvement is substantial (~10% in accuracy) for the more challenging CIFAR- 100 with high noise ratios. Appendix A shows comparison with more methods in T The results on CIFAR- 10 with asymmetric noise is shown in Table We use 40% because certain classes become theoretically indistinguishable for asymmetric noise larger than 50%.
Method
Best
Last
Cross- Eotso-E
85.0
72.3
F-correction Patrini et al.
2017
87.2
83.1
M-correction (Arazo
2019
87.4
86.3
Iterative-CV Chen
2019)
88.6
88.0
P-correction
Wu
2019)
88.5
88.1
Joint-Optim
Tanaka ci al. 2018
88.9
88.4
Meta-Learning (Li et al.
2019
89.2
88.6
DivideMix
93.4
92.1
Table 2: Comparison with state-of-the-art methods in test accuracy (%) CIFAR-10 with 40% asymmetric noise. We re-implement all methods under the same setting.
Table and Table show the results on Clothing 1M and Web respectively. DivideMix consistently outperforms ofate-of-the-arn methods across all datasets with different types of label noise. For WebVision, we achieve more than 12% improvement in top-1 accuracy.
7

---

Published as a conference paper at ICLR 2020
Method
Test Accuracy
Cross-Entropy
69.21
F-correction (Patrini 2017)
69.84
M-correction Arazo ot al.
2019
71.00
Joint- Optim
(Tanaka ct al.
2018
72.16
Meta-Cleaner 7/hang et al
2019)
72.50
Meta-Learning
2019)
73.47
P-correction (Yi Wu
2019)
73.49
DivideMix
74.76
Table 3: Comparison with state-of-the-art methods in test accuracy (%) on Clothing MM. Results for baselines are copied from original papers.
Web Vision
ILSVRCI2
Method
top1
top5
top1
top5
F-correction Patrini et al.
2017
61.12 12
82.68
57.36
82.36
Decoupling (Malach & Shalcv-Shward D2L (Ma el at. 2018)
2017)
62.54
84.74
58.26
82.26
62,68
84.00
57.80
81.36
MentorNet (Jang al.
2018
63.00
81.40
57.80
79.92
Co-teaching (Han
2018
63.58
85.20
61.48
84.70
lterative-CV CChen
2019
65.24
85.34
61.60
84.98
DivideMix
77.32
91.64
75.20
90.84
Table 4: Comparison with state-of-the-ard methods trained on (mini) Web WebVision dataset. Numbers denote top-1 (top-5) accuracy (%) on the WebVision validation and the ImageNet ILSVRCI2 validation set. Results for baseline methods are copied from Chen et al. (2019).
4.3 ABLATION STUDY
We study the effect of removing different components to provide insights into what makes DivideMix successful. We analyze the results in Table as follows. Appendix C contains additional explanations.
Dataset
CIFAR-IO
CIFAR- 100
Noise type
Sym.
Asym.
Sym.
Methods/Noise ratio
20%
50%
80%
90%
40%
20%
50%
80%
90%
Best
96.1
94.6
93.2
76.0
93.4
77.3
74.6
60.2
31.5
DivideMix
Last
95.7
94.4
92.9
75.4
92.1
76.9
74.2
59.6
DivideMix with ((2) test
Best Last
95.2
94.2
93.0
75.5
92.7
75.2
72.8
58.3
29.9
95.0
93.7
92.4
74.2
91.4
74.8
72.
57.6
29.2
DivideMix w/o co-training
Best
95.0
94.0
92.6
74.3
91.9
74.8
56.7
27.7
Last
94.8
93.3
92.2
73.2
90.6
74.1
71.7
56.3
27.2
DivideMix w/o label refinement
Best
96.0
94.6
73.7
87.7
76.9
58.7
Last
95.5
94.2
92.7
73.0
86.3
76.4
73.9
58.2
26.3
Best
95.3
94.1
92.2
73.9
89.5
76.5
73.1
58.2
26.9
DivideMix w/o augmentation
Last
94.9
93.5
91.8
73.0
88.4
76.2
72.6
58.0
26.4
Best
94.1
92.8
89.7
70.
86.5
73.7
70.5
55.3
25.0
Divide and MixMatch
Last
93.5
92.3
89.1
68.6
85.2
72.4
69.7
53,9
23.7
Table 5: Ablation study results in terms of test accuracy (%) on CIFAR- 10 and CIFAR-100.
study the effect of model ensemble during test, we use the prediction from a single model ((1) instead of averaging the predictions from both networks in DivideMix. Note that the training process remains unchanged. The decrease in accuracy suggests that the ensemble of two diverged networks consistently yields better performance during inference.
To study the effect of co-training, we train single network using self-divide (i.e. divide the training data based its loss). The performance further decreases compared to ((1).
8

---

Published as a conference paper at ICLR 2020
We find that both label refinement and input augmentation beneficial for DivideMix.
We combine self-divide with the original MixMatch as naive baseline for using SLL in LNL.
Appendix A also introduces more in-depth studies in examining the robustness of our method to label noise, including the AUC for clean/noisy sample classification on CIFAR-10 10 training data, qualitative examples from Clothing IM where our method can effectively identify the noisy samples and leverage them as unlabeled data, and visualization results using t-SNE.
5 CONCLUSION
In this paper, we propose DivideMix for learning with noisy labels by leveraging SSL. Our method trains two networks simultaneously and achieves robustness to noise through dataset co-divide, label co-refinement and co- guessing. Through extensive experiments across multiple datasets, we show that DivideMix consistently exhibits substantial performance improvements compared state-of-the- methods. For future work, we are interested in incorporating additional ideas from SSL to LNL, and vice versa. Furthermore, we are also interested in adapting DivideMix to other domains such as NLP.
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O' D'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, pp. 312-321, 2019.
Devansh Arpit, Stanislaw Jastrzkebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look at memorization in deep networks. In ICML, pp. 233-242, 2017.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurlPS, 2019.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pp. 1062-1070, 2019.
Yifan Ding, Liqiang Wang, Deliang Fan, and Boqing Gong. semi-supervised two-stage approach to learning from noisy labels. In WACV, pp. 1215-1224, 2018.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS, pp. 529-536, 2004.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, pp. 8536-8546, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, pp. 630-645, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NewrlPS, pp. 10477-10486, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. ICMI, pp. 2309-2318, 2018.
Kycongbo Kong, Junggi Lee, Youngchul Kwak, Minsung Kang, Scong Gyun Kim, and Woo-Jin Song. Recycling: Semi-supervised learning with noisy labels in deep neural networks. IEEE Access, 7:66998 2019.
Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. ICML, pp. 1888-3498, 2019.

---

Published as a conference paper at ICLR 2020
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Mater's thesis, University of Toronto, 2009.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification,, object detection, and visual relationship detection at scale. arXiv:1811.00982. 2018.
Samuli Laine and Timo Aila. Temporal ensembling for semi semi-super learning, In ICLR, 2017.
Dong-Hy Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning, volume 3, pp. 2, 2013.
Kuang-Huee Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In CVPR, pp. 5447-5456, 2018.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Learning to learn from noisy labeled data. In CVPR, pp. 5051-5059, 2019.
Wen Li, Limin Wang. Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv:/708.02880 2017a.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, pp. 1928-1936, 2017b.
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Sudanthi N. R. Wijewickrema, and James Bailey. Dimensionnlity-driven learning with noisy labels. In ICML, pp. 3361-3370, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, pp. 185-201, 2018
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In NIPS, pp. 960-970, 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 1179):1979-1993, 2019.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: loss correction approach. In CVPR, pp. 2233-2241, 2017.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR Workshop, 2017.
Haim H. Permuter, Joseph M. Francos, and Ian Jermyn. study of gaussian mixture models of color and texture features for image classification and segmentation. Pattern Recognition, 39(4) 695-706, 2006.
Scott F. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR, 2015.
Mengye Ren, Wenyuan Zeng, Bin Yang. and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331-4340. 2018.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In ICMI., pp. 5739-5748, 2019.
10

---

Published as a conference paper at ICLR 2020
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-ye v4, inception-resner and the impact of residual connections on learning. In AAAI, pp. 4278 4284, 2017.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pp. 5552-5560, 2018.
Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C. Alexander, and Nathan Sil- berman. Learning from noisy labels by regularized estimation of annotator confusion. In CVPR, 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NIPS, pp. 1195-1204, 2017.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff' A. Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In ICML, pp. 6234-6243, 2019.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks In NIPS, pp. 5601-5610, 2017.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge J. Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, pp. 6575-6588 2017.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, pp. 8688 8696, 2018.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In CVPR, 2019.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? ICML, pp. 7164-7173, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Hongyi Zhang. Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Weihe Zhang, Yali Wang, and Yu Qiao. Metacleaner: Learning to hallucinate clean representations for nobse-labeled visual recognition. In CVPR, 2019.

---

Published as a conference paper at ICLR 2020
APPENDIX A ADDITIONAL EXPERIMENT RESULTS
In Table 6 we compare DivideMix with previous methods under the symmetric noise setting where true labels cannot be maintained. DivideMix significantly outperforms previous methods which use deeper or wider network architectures.
CIFAR-10
CIFAR-100 1IFAR-1000
Method
Architecture
20%
40%
60%
80%
20%
40%
60%
80%
MentorNet Jiang et al.
2018
WRN-101
92.0
89.0
49.0
|
73.0
68.0
35.0
D2L (Ma al. 2018
CNN-12/RN-44
85.1
83.4
72.8
62.2
52.0
42.3
Reweight (Ren al. 2018
WRN-28
86.9
61.3
Abstention Thulasidasan et al. 2019]
WRN-28
93.4
90.9
87.6
70.8
75.8
68.2
59.4
34.1
DivideMix
PRN-18
96.2
94.9
94.3
79.8
77.2
75.2
72.0
60.0
Table 6: Comparison with state-of-the-arr methods in test accuracy (%) on CIFAR- 10 and CIFAR-100 with symmetric noise. Numbers are copied from original papers. Key: WRN (Wide ResNet), PRN (PreActivation ResNet). DivideMix outperforms previous methods that use deeper/wider networks.
In Figure we show the Area Under Curve (AUC) for clean/noisy sample classification CIFAR- 10 training data from one of the GMMs during the first 100 epochs. Our method can effectively separate clean and noisy samples as training proceeds, even for high noise ratio.
0.98
0.96
AUC
0.94
0.92
20% label noise
label noise
0.9
80% label noise
0.88
10
20
30
40
50
60
70
80
90
100
Epoch
Figure 3: Area Under Curve for clean/noisy image classification CIFAR-10 10 training samples. Our method can effectively filter out noisy samples and leverage them as unlabeled data.
In Figure 4
we show example images in Clothing MM identified by our method as noisy samples. Our method achieves noise filtering by discarding the noisy labels (shown in red) and using the co-guessed labels (shown in blue) to regularize training.
Vest
Hoodie
Suit
Dress
Downcoat
Underwear
Dress
Knitwear
Dress
Windbreaker
Chiffon
T-shirt
Figure 4: Clothing M images identified as noisy samples by method. Ground-truth labels are shown above red and co- guessed labels are shown below blue.
12

---

Published as a conference paper at ICLR 2020
visualize the features of training images using t-SNE (Maaten & Hinton 2008 The model is trained using DivideMix for 200 epochs on CIFAR-10 with 80% label noise. The embeddings form 10 distinct clusters corresponding to the true class labels, not the noisy training labels, which demonstrates our method's robustness to label noise.
In Figure
(a) Noisy training labels.
(b) True labels.
Figure 5: T-SNE of training images after training the model using DivideMix for 200 epochs on CIFAR-10
with 80% label noise. Different colors indicate (a) noisy training labels or (b) true labels. DivideMix is able to learn the true class distribution of the training data despite the label noise.
APPENDIX B ADDITIONAL TRAINING DETAILS
For CIFAR experiments, the only hyperparameter that we tune on a per-experiment basis is the unsupervised loss weight Au. Table shows the value that we use. A larger A is required for stronger regularization under high noise ratios with more classes.
For both Clothing MM and Web we use the same set of hyperparameters M = 2, T 0.5, 0.5, and train the network using SGD with a momentum of 0.9, weight decay of 0,001, and a batch size of 32. The warm up period is epoch. For Clothing IM, we train the network for epochs. The initial learning rate is set as 0.002 and reduced by factor of 10 after 40 epochs. For each epoch, we sample 1000 mini-batches from the training data while ensuring the labels (noisy) are balanced. For WibVision, we train the network for 100 epochs. The initial learning rate is set as 0.01 and reduced by factor of 10 after 50 epochs.
0.5, A,
CIFAR-10
CIFAR-100
Hyperparameter
Asym. 40%
20%
50%
80%
90%
20%
50%
80%
90%
A
25
150
150
Table 7: Unsupervised loss weight Au for CIFAR experiments. Higher noise ratio requires stronger regularization from unlabeled samples.
APPENDIX C ADDITIONAL EXPLANATIONS FOR ABLATION STUDY
Here we clarify some details for the baseline methods in the ablation study. First, DivideMix w/o co-training still has dataset division, label refinement and label guessing, but performed by the same model. Thus, the performance drop (especially for CIFAR-100 with high noise ratio) suggests the disadvantage of self-training. Second, label refinement is important for high noise ratio because more noisy samples would be mistakenly divided into the labeled set. Third, augmentation improves performance through both producing more reliable predictions and achieving consistency regularization. In addition, same as Berthelot et al. (2019), we also find that temperature sharpening essential for our method perform well.
13

---

Published as a conference paper at ICLR 2020
APPENDIX D TRAINING TIME ANALYSIS
We analyse the training time of DivideMix to understand its efficiency. In Table we compare the total training time of DivideMix on CIFAR-IO with several state-of-thearr methods, using single Nvidia V100 GPU. DivideMix is slower than Co-teaching+ (Yu et al. 20119), but faster than
P-correction Yi Wu 2019) and Meta-Learning (Li et al. 2019) which involve multiple training iterations. In Table 9, we also break down the computation time for each operation in DivideMix.
Co-teaching+"
P-correction
Meta-Learning
DivideMix
h
6.0
8.6
5.2
Table 8: Comparison of total training time (hours) CIFAR-10.
Co-Divide (Alg.
line 4-8)
Data MixMatch (Alg
line 12-24)
Berward-Backweer (Alg.
line 25-27)
1
16.0
12.5
Table 9: Computation time (seconds) per-epoch for each operation in DivideMix during training.
14


Reviews:
[{"review_id": "SyxNO3Ne5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a DivideMix framework for learning with noisy labels, where they first Co-Divide the training data into a labeled clean set and an unlabeled noisy set by modeling the per-sample loss distribution with GMM and using the small loss trick, then they exploit MixMatch to train the model on those labeled and unlabeled data in a semi-supervised manner. Experiments and comparisons with SOTA are provided, together with an ablation study.\n\nPros:\n-The paper bridges the area of learning with noisy labels with semi-supervised learning and proposes an interesting method to learn with noisy labels in a semi-supervised manner. The treatment proceeds from analyzing good unsupervised loss functions, improving the MixMatch and implementing thorough experiments.\n\n-The paper is clear and flows smoothly. The treatment is thorough, proceeding from designing algorithms, implementing experiments and an ablation study.\n\n-The impact of the method is a clear asset. Most existing label noise works focus on certain noise models, but this paper is more general and proposes an interesting method to learn with noisy labels in a semi-supervised manner, and the experimental results are promising.\n\n-The effort made on designing the confidence penalty for asymmetric noise is interesting.\n\nRemarks:\n-Sec 3.1: it seems that \\tau is an important hyperparameter for dividing the noisy data into labeled and unlabeled sets, but in Sec 4 I only see that \\tao is set to be 0.5 or 0.6 for CIFAR, what about other experiments? I’m also wondering if \\tao needs some decay during training? For example, it may be 0.5 at early training epochs, but may be smaller at last, since deep NNs are gradually fitting noisy features during training.\n\n-Algorithm: the proposed method needs to train two networks simultaneously. During each epoch, it firstly divides the noisy data by modeling the per-sample loss distribution with GMM, and then do MixMatch with label co-refinement and co-guessing. I’m a bit concerned about efficiency. So how about the computation time?\n\n-Experiments: more details on experimental protocol may be needed: what kind of hyperparameter tuning was done? How many repeated runs? It would be helpful to report the means and standard deviations based on repeated samplings.\n\nOverall take: this paper proposes a thorough treatment of learning with noisy labels in a semi-supervised manner, designing the algorithm and testing it empirically, which is an interesting and important contribution. My only concern is about the novelty since the small loss trick in label noise and the MixMatch approach in SSL are already explored by many recent studies, but to the best of my knowledge, this paper is the first to unify them to solve label noise problems.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1gKsV9EYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method named DivideMix for learning with noisy labels, on top of the recent semi-supervised learning method MixMatch from Google. The idea is to model the per-sample loss distribution with a mixture model to dynamically DIVIDE the training data into (a labeled set with clean samples) and (an unlabeled set with noisy samples) and trains the model on both the labeled and unlabeled data in a semi-supervised manner.\n\nThe novelty is borderline. In the area of learning with noisy labels, it is known that SSL can work under this problem setting for several years, for example, the famous method \"virtual adversarial training\" from ICLR 2016 and a recent method \"smooth neighbors on teacher graphs\" from CVPR 2018. As a consequence, it is not surprising that the latest MixMatch can work as well, since MixMatch comes from mixup, virtual adversarial training and entropy minimization. This makes the novelty borderline. However, the significance may still be high according to the reported experimental results, and thus we may accept it to let more deep learning practitioners see the promising results.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HkeBKNcaKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm that learns with noisy labels that achieves state-of-the-art results. Their algorithm tries to exploit the noisy samples by assigning a ‘correct’ label through MixMatch. It borrows the idea from both semi-supervised learning and learning with label noise. \n\nSuggestions:\n\n1. When the author talked about “correct the loss function”, there are in fact two types of corrections: the first type tries to correct the loss function by equally treating all the samples, e.g., classical Huber loss, or F-correction. The second type tries to either re-weight samples or separate clean and noisy samples explicitly, which also results in correcting the loss function. It would be more clear if Section 2.1 can emphasize the difference between the two. \n\n2. Also, there are other papers providing different but insightful ideas to label noise problem. The authors addressed the idea of using co-learning to avoid confirmation bias. However, it should be noted that this is not the only way to avoid confirmation bias, and their are other methods without using two networks [1-2], both providing some theoretical insights to the problem. It would be good to include them in the related work as well. \n\n3. I would like to see a comparison of running time other than the accuracy, understanding the efficiency of each algorithm is important from a practical perspective. \n\n[1] Learning with Bad Training Data via Iterative Trimmed Loss Minimization, Yanyao Shen, Sujay Sanghavi, ICML 2019.\n[2] Robust Learning from Untrusted Sources,Nikola Konstantinov, Christoph H. Lampert, ICML 2019\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an algorithm for noisy labels by adopting an idea in the recent semi-supervised learning algorithm.

As two problems of training noisy labels and semi-supervised ones are closely related, it is not surprising to expect such results as pointed out by reviewers. However, reported thorough experimental results are strong and I think this paper can be useful for practitioners and following works. 

Hence, I recommend acceptance.
Decision: Accept (Poster)