Title: IEG: ROBUST NEURAL NETWORK TRAINING TO TACKLE SEVERE LABEL NOISE

Abstract: Collecting large-scale data with clean labels for supervised training of neural networks is practically challenging. Although noisy labels are usually cheap to acquire, existing methods suffer severely for training datasets with high noise ratios, making high-cost human labeling a necessity. Here we present a method to train neural networks in a way that is almost invulnerable to severe label noise by utilizing a tiny trusted set. Our method, named IEG, is based on three key insights: (i) Isolation of noisy labels, (ii) Escalation of useful supervision from mislabeled data, and (iii) Guidance from small trusted data. On CIFAR100 with a 40% uniform noise ratio and 10 trusted labeled data per class, our method achieves 80.2±0.3% classification accuracy, only 1.4% higher error than a neural network trained without label noise. Moreover, increasing the noise ratio to 80%, our method still achieves a high accuracy of 75.5±0.2%, compared to the previous best 47.7%. Finally, our method sets new state of the art on various types of challenging label corruption types and levels and large-scale WebVision benchmarks.

1. INTRODUCTION:

Training deep neural networks usually requires large-scale labeled data. However, the process of data labelling by humans is challenging and expensive in practice, especially in domains where expert annotators are needed such as medical imaging. A great number of methods have been proposed to train neural networks from datasets with noisy labels due to cheap acquisition (e.g. loosely-controlled procedures, crowd-sourcing, web search, text extraction, etc) (Zhang & Sabuncu, 2018) . However, deep neural networks have high capacity for memorization. When noisy labels become prominent, neural networks inevitably overfit to noisy labeled data (Zhang et al., 2017a; Tanaka et al., 2018) .

To overcome this problem, we argue that rethinking training dataset construction along with model training is necessary. Most methods consider a setting where the entire training dataset is acquired using the same labeling technique. However, when real-world constraints such as the labeling budget are considered, it is often practically feasible to also construct a tiny dataset that contains highlytrusted clean labels. If methods based on this setting can demonstrate high robustness even with extremely noisy labels, new horizons can be opened in training data labeling practices. There are a few recent methods that demonstrate strong performance by leveraging a small trusted dataset and training on a large noisy dataset, including learning weights of training data (Jiang et al., 2018; Ren et al., 2018) , loss correction (Hendrycks et al., 2018) , and knowledge graph (Li et al., 2017b) . However, these methods still require a substantially large trusted set to reliably yield high performance. We show that it is possible to significantly reduce the necessary size of the trusted set while maintaining superior performance when suitable regularization is used (e.g. some methods use 10% of the total training data while our method only uses 0.2%).

In this paper, we consider three key factors and demonstrate a new method towards a noise robust neural net training strategy:

• Isolation: Reweigh training samples to isolate noisy labeled data and prevent mislabeled data from misleading neural network training. • Escalation: Escalate supervision from mislabeled data via pseudo labels to make use of information in mislabeled data. Semi-supervised Fully-supervised SoTA (Noisy labels) IEG (Proposed)

Figure 1 : Image classification results on CIFAR100 showing the benefit of IEG. IEG denotes our method which outperforms semi-supervised learning methods at up to a 95% noise ratio. Fullysupervised is trained with all labeled clean data. Semi-supervised is our extension of IEG for semisupervised setting, which has the best reported results. SoTA (noisy labels) denotes the previous best results for noise robustness (50 trusted data per class are used). The noise ratio of random label assignment is 0.99. 10 trusted labeled data per class are available for Semi-supervised and IEG. See Section 3.4 for more details.

• Guidance: Use a tiny trusted dataset for guided training with strong regularization to prevent overfitting.

Although previous work has attempted to deal with some of these factors, the performance gains have been moderate. Ideally, even with a small amount of correct labels in the noisy training data, a robust learning method should distill that information into model training and outperform the semi-supervised learning scenario where labels are completely ignored. However, from a view of comparison to state-of-the-art semi-supervised learning methods (Verma et al., 2019; Berthelot et al., 2019) in Figure 1 (explained in experiments), we observe that state of the art in noise-robust learning is inferior even with a 50% noise ratio (i.e. they cannot optimally distill the valuable supervised signal from almost half of the data), suggesting there is significant amount of room for improvement. This raises two important questions: 1. Should we discard noisy labels and opt in for semi-supervised training at high noise regimes? 2. How can we better distill the useful knowledge in noisy labels?

Contributions: First, we present a novel training method, named IEG, that tackles the above two questions effectively in an unified framework. IEG is designed to be model-agnostic and to generalize to any type of label corruption. Figure 1 demonstrates that even with extremely noisy labels (as high as 95%), our method is almost invulnerable to severe noise. We achieve the goal by addressing the three key factors effectively with the following complementary objectives:

1. A meta learning based re-weighting and re-labelling objective to simultaneously learn to weigh the per-datum importance and progressively escalate supervised losses of training data using pseudo labels as a replacement of original labels.

2. A label estimation objective to serve as the initialization of the meta re-labelling step and escalate supervision from mislabeled data.

3. An unsupervised regularization objective to enhance label estimation and improve overall representation learning.

Second, our method sets new state of the art on CIFAR10, CIFAR100 and on the large-scale Web-Vision, in many label corruption types by a large margin.

2. METHOD 2.1 META OPTIMIZATION-BASED REWEIGHING AND RELABELING:

We leverage the meta optimization to automatically 1) estimate the weight of each data point when it is possibly mislabeled and 2) choose between pseudo labels and original labels when pseudo labels make useful contributions to the trusted set performance.

Given a dataset of N inputs with noisy labels D u = {(x i , y i ), 1 < i < N } and also a small dataset (denoted as probe data) of M of samples with trusted labels D p = {(x i , y i ), 1 < i < M }, where M N . The main idea of learning-to-reweight (L2R) (Ren et al., 2018) is training neural networks with a weighted cross-entropy loss for each training batch of size B:

EQUATION

where ω is a vector that its element ω i gives the weight for the loss of a training pair. Φ(•; Θ) is the targeting neural network that outputs the class probability and N i=1 L(y i , Φ(x i ; Θ)) is the standard softmax cross-entropy loss for each training data pair (x i , y i ). Note that Θ is a function of ω, but we omit frequently for conciseness.

Treating ω as learnable parameters, the meta step behaves like a probe to seek for the optimal ω for each training data in D u such that the trained model using equation 1 can obtain the best performance on the trusted data D p . However, it is computationally-costly to find optimal ω * since each update step requires training the model until converge to get Θ. In practice, we can use an online approximation (Ren et al., 2018; Finn et al., 2017) to perform a single meta gradient-descent step

Θ t+1 = Θ t -α∇ Θ N i ω i L y i , Φ(x; Θ t )

, where α is the step size,

EQUATION

We expect that the optimized ω * should assign almost-zero value to mislabeled data to isolate mislabeled data from clean data. Optimization of ω is based on back-propagation with second-order derivatives.

When the noise ratio is high, a significant amount of data would be discarded. To address this information loss, we propose to generalize the meta update step to use information from the discarded data through a pseudo-labeling strategy. Given the function of pseudo label estimator g(x, Φ) (introduced in the next section), we generalize the meta optimization with the following objective to utilize both the data weighting and re-labelling:

EQUATION

The optimized re-labelling controller λ * t is updated based on the sign of its gradient,

EQUATION

where β is the step size. L p is computed on a batch sampled from D p . The reweighing controller ω * = ω -∇ω can be obtained in the similar way. We use the sign of the gradient |∇λ| instead of λ 0 -∇λ because 1) ∇λ would get very small when pseudo labels are close to real labels (please see Appendix) and 2) simply averaging y i and g(x i , Φ) using scalar λ * makes resulting pseudo label distribution less sharp.

After the meta step, we compute two cross-entropy losses given respective optimal values,

L ω * = N i ω * i L λ 0 y i , Φ(x i ; Θ t ) , L λ * = 1 N N i L ỹi , Φ(x i ; Θ t ) , ỹ = y i , if λ * t,i > 0 g(x i , Φ), otherwise

(5) where B is the batch size. Similar to L2R, we use momentum SGD for model training. We compute the meta step model parameters Θ t+1 by calculating the exact momentum update using momentum value of the SGD optimizer at each optimization step1 .

2.2. ESCALATING SUPERVISION FROM MISLABELED DATA:

Given learned data weights ω at a training step, IEG separates the data as either possibly-mislabeled or possibly-clean using the binary criterion I(ω i < T ) , where T is a scalar threshold. IEG utilizes mislabeled data with pseudo labels and probe data with trusted labels to provide training supervision with extra regularization, in order to achieve promised Escalation and Guidance.

2.2.1. PSEUDO LABELS:

Utilizing the pseudo labels from unlabeled training data is widely studied for semi-supervised learning, by converting predictions to one-hot label (Lee, 2013) or their smoother versions (Tanaka et al., 2018; Lee, 2013) ).

Neural network predictions can be unstable to input perturbations (Zheng et al., 2016; Azulay & Weiss, 2018) . Enforcing consistency in neural network predictions has been shown to be important for model performance in semi-supervised learning (Xie et al., 2019) . Therefore, if perturbations of an input obtain diverse model predictions, we should not trust the predictions to be pseudo labels. Recently-proposed state of the art semi-supervised learning method MixMatch (Berthelot et al., 2019) considers this principle in its design. We adopt this approach to compute g(x i , Φ) in IEG. The resulting averaged predictions are given by: g i (x, Φ) = P r

1 τ i / i P r 1 τ i , where P r = 1 K Φ(x) + K-1 k Φ(x k )

, where xk is k-th randomly augmented version of input x. g leads to a soft pseudo label. g i is i-th class of the pseudo label. τ is a softmax temperature scaling factor (τ = 0.5 in this paper).

2.2.2. REGULARIZATION TO ENABLE GUIDANCE USING PROBE DATA:

When noise ratio is high, even though the meta step effectively prevents misleading optimization (i.e. most elements in ω are zero), we potentially waste a lot of useful supervision to maintain high-performance learning. Therefore, we seek extra ways to improve supervision. Here, we show that even very small amount of trusted labeled data can improve model performance significantly. We aim to leverage the information from probe data, besides its original use for meta optimization. An appropriate regularization is critical for this purpose, otherwise the neural network would quickly overfit to the small probe data and yield ineffective meta gradients for learning ω and λ (e.g. equation 4).

To this end, we adopt the MixUp regularization and construct extra supervision losses using the data in the form of convex combinations using the data and their labels given a mixup factor β: Mix β (a, b) = βa + (1 -β)b, β ∼ Beta(0.5, 0.5). It has been shown effective in recent semisupervised learning methods (Hataya & Nakayama, 2019; Verma et al., 2019) . In detail, for each data x a in the concatenated data pool in D p ∪ Du ∪ D u , we apply pairwise MixUp between the input batch and its random permutation,

EQUATION

)

where Du is the augmented copy of D u . We introduce two softmax cross-entropy losses, L p β for resulting mixed data when x a ∼ D p is from probe data) and L u β when x a ∼ Du ∪ D u . We show that this strategy of IEG reduces the probe data size to be as small as one sample per class.

2.2.3. PSEUDO LABELS NEED CONSISTENT PREDICTIONS:

Ideal pseudo labels should be close to real labels. The pseudo labels of IEG are generated by averaging the predictions over augmentations. However, if the predictions are controversial to each other, their contributions would cancel out and yield flattened average outputs. Consequently, the supervision using these pseudo labels would not encourage the model to be discriminative. In our insight, to generate more sharper pseudo outputs, reducing the controversy of augmentations is necessary. Therefore, we propose to encourage discriminability of the pseudo labels via enforcing consistency. The KL-divergence objective IEG used is defined as

EQUATION

Algorithm 1 summarizes a training step and presents all objectives along with their loss coefficients. 1 Generate the augmentation Xu of X u . 2 Estimate the persudo labels via g(x u , Φ), x u ∼ X u ∪ Xu (Section 2.2.1).

3 Compute optimal λ * and ω * via the meta step (Section 2.1). 4 Split the training batch X u (also corresponding Xu ) to possible clean batch X c u and possible mislabeled batch X u u using the binary criterion I(ω * < T ). 5 Compute the mixup of joint batch set (Section 2.2.2),

X p ∪ X u u ∪ X c u ∪ Xu u ∪ Xc u

where Xu u ∪ X u u uses persudo labels estimated by g(•, Φ). 6 Compute the total loss for model update

L ω * + L λ * + L p β + p L u β + k L KL , .

7 Conduct one step stochastic gradient descent to obtain Θ t+1 .

3. EXPERIMENTS:

We validate the proposed IEG method on multiple datasets (CIFAR10, CIFAR100, and large-scale WebVision datasets) with various kinds of common label corruptions (including uniform and semantic types). We also conduct extensive ablation studies to demonstrate the key aspects of IEG.

3.1. EMPIRICAL TRAINING DETAILS:

Here we discuss key training details and hyperparameters, that are shown to be beneficial for our experiments.

Learning rate decay: We adopt the Cosine learning rate decay with warm restarting2 . We observe 3%-5% accuracy improvement on CIFAR datasets, especially at large noise ratios. Figure A1 of Appendix plots the curves. Although it works particularly well in IEG, we do not observe strong benefit either training standard neural networks or training L2R, also not in recent literature (Gotmare et al., 2019) and (Song et al., 2018) which uses cosine learning rate.

Model selection: Although the size of probe data is small, we find our method less likely memorizes the probe data. So it can be potentially be monitored as validation set for model selection. Loshchilov & Hutter (2017) also indicates the needless of the validation set for model selection with cosine learning rate decay. Therefore, we directly select models at the lowest learning rate before 200 epochs. Augmentation: The purpose of augmentation is to generate pixel perturbation around the original training data. We adopt the AutoAugment (AA) technique for image data (Cubuk et al., 2018) to achieve this, which including operations (learned policy augmentation →flip→random crop→cutout (DeVries & Taylor, 2017)). In detail, for each input image, we first generate one standard augmentation (random crop and horizontal flip) and then apply AA to generate K random augmentations on top of the standard one. We use K = 2 augmentations in our experiments.

3.2. CIFAR NOISY LABEL EXPERIMENTS:

For all CIFAR experiments, we set T = 1, p = 5, k = 20. The models are trained on a single NVIDIA v100. std of reported results are obtained by 3 runs with random seeds. We compare the proposed IEG method against several recent methods, which achieve leading performance in public benchmarks. Similar to L2R, we use the Wide ResNet (WRN28-10) (Zagoruyko & Komodakis, 2016) as default, unless specified otherwise, for fair comparison.

Common random label noise: Table 1 compares the results for CIFAR10 with uniform noise ratios of 0.2, 0.4, and 0.8. 10 probe images per class are used. We also test IEG using ResNet293 , which is much smaller than ones used by compared methods. Using WRN28-10, IEG leads to 96.5% accuracy with 20% noise ratio and 94.7% accuracy with 80% noise ratio, demonstrating nearly noise-free performance. IEG still achieves the best performance with ResNet29. We also train IEG with 0% noise as reference. We observe most results even outperforms the results with standard training of WRN28-10/ResNet29 (see captions of Table 1 ). This shows that our proposed method provides additional form of regularization to improve generalization. Table 2 compares the results in CIFAR100 with uniform noise ratios of 0.2, 0.4, and 0.8. We also report results given 10 images, 5 images and the extreme case of 1 image per class for probe data, much lower than the other methods use. IEG significantly outperforms existing methods. The improvement is remarkable at higher noise ratios.

Three types of semantic label noise: Next, we test IEG on more realistic noisy settings on CIFAR. 10 images per class are used as probe data. ric noise ratios of 0.2, 0.4, and 0.8. Asymmetric noise is known as a more realistic setting because it corrupts semantically-similar classes (e.g. truck and automobile, bird and airplane) (Patrini et al., 2017) . Moreover, we follow RoG (Lee et al., 2019) to generate semantic noisy labels by using a trained VGG-13 (Simonyan & Zisserman, 2015) (the hardest setting) on 5% of CIFAR10 and 20% of CIFAR100 (we directly use the data provided by the author). Table 4 reports the compared results.

Lastly, we test IEG on three kinds of open-set noisy labels (this setting replaces images to out-ofdistribution images of the same labels (Wang et al., 2018) ) in Table A1 of Appendix. In all semantic noise settings, IEG consistently outperforms the compared methods by a significant margin.

3.3. WEBVISION REAL-WORLD NOISY LABEL EXPERIMENTS:

WebVision (Li et al., 2017a ) is large-scale dataset which reflects real-world noisy labels as their images are obtained by crawling from the Flickr website and Google Images Search using the labels. It contains 2.4 million images and shares the 1000 classes with ImageNet (Deng et al.) . We also follow (Jiang et al., 2018) to create a mini version of WebVision, which includes the Google subset images of the top 50 classes. To create the probe data, we set aside 10 images per class from the ImageNet training data. We train models using the WebVision training set on a Google Cloud TPU and evaluate on the ImageNet validation set. We set T = 1, p = 4, k = 8. Table A2 compares the results. While the compared methods use a larger InceptionResNetv2 (IRv2), we use ResNet-50 due to its memory efficiency, albeit the lower expected performance due to its lower capacity. We verify the performance of backbone neural networks. We observe slight (<0 

3.4. COMPARISON TO SEMI-SUPERVISED LEARNING:

We compare IEG to the state-of-the-art semi-supervised learning method MixMatch Berthelot et al. (2019) to verify how much useful information IEG can distill from mislabeled data. The unsupervised components which IEG incorporated can be also applied onto semi-supervised methods. We simply remove the Isolation (meta re-weighting and re-labelling) of IEG and treat all training data with noisy labels as unlabeled to enable semi-supervised training (denoted the resulting method as EG). Figure 1 shows the comparisons and Table 6 reports the detailed results. IEG improves the performance largely given the 80% label noise ratio. In addition, EG demonstrates remarkable benefits, for example, from 34.5% to 57.6% on CIFAR100. It demonstrates that it is necessary to enforce consistency for pseudo labels for better discriminability.

3.5. ABLATION STUDIES AND DISCUSSIONS:

Here we study the individual objective components of IEG and their effectiveness. 

4. RELATED WORK:

Reweighing training data has been shown to be effective (Liu & Tao, 2015) . However, estimating effective weights is challenging. Ren et al. (2018) proposes a meta learning approach to directly optimize the weights in pursuit of best validation performance. Jiang et al. (2018) alternatively uses teach-student curriculum learning to weigh data. (Han et al., 2018) uses two neural networks to cotrain and feed data to each other selectively. Another direction is modeling confusion matrix for loss correction, which has been widely studied (Sukhbaatar et al., 2014; Natarajan et al., 2013; Tanno et al., 2019; Patrini et al., 2017; Arazo et al., 2019) . For example, (Hendrycks et al., 2018) shows that using a set of trusted data to estimate the confusion matrix has significant gains.

The approach on relabeling corrupted samples is another direction (Li et al., 2017b; Tanaka et al., 2018; Veit et al., 2017; Han et al., 2019) . Along this, Reed et al. (2014) uses bootstrapping to generate new labels. (Li et al., 2019) leverage the meta learning framework to verify multiple label candidates before doing actual training. Relabeling is similar to the pseudo label approach in semisupervised learning (Lee, 2013) . Besides pseudo labels, building connections to semi-supervised learning has been recently expanded (Kim et al., 2019) , which applies semi-supervised losses to improve representation learning from mislabeled data. For example, Hataya & Nakayama (2019); Arazo et al. (2019) uses Mixup (Zhang et al., 2017b) to augment data and demonstrates clear benefits. Ding et al. (2018) ; Kim et al. (2019) identifies mislabeled data first and then leverages semisupervised techniques.

5. CONCLUSION:

In this paper, we present a robust and generic neural network training method to overcome severe label noise. Our method, named IEG, is based on unification of the mechanisms to isolate the noise labels via meta optimization, escalate the supervision mislabeled data via pseudo meta re-labeling, and effectively use small trusted data to guide training. IEG demonstrates significant and consistent improvements over previous state of the art methods on common benchmarks. If y i and Φ(x i ; Θ) are close to each other around Θ t , the derivative ∂ ∂λi,t E L p λ=λ0,ω=ω0 ] would be close to 0. Thus, for a converged model with low training error, the amount of update on λ would be close to zero.

:

We set initial the values as ω0 = 1/B and λ0 = 0.9 due to better performance, observed empirically.

We set the initial cycle length to be one epoch, and after then cycle length increases by a factor of 1.5 and meanwhile the restart learning rate decreases by a factor of 0.9 as described in(Loshchilov & Hutter, 2017)

For ResNet29 we use in this paper, we follow this pre-activation (v2) implementation https: //github.com/keras-team/keras/blob/master/examples/cifar10_resnet.py, which contains 0.84M parameters.

Figures and Tables:

A training step of IEG at time step t Input: Current model parameters Θ t , A batch of training data X u from D u , a batch of probe data X p from D p , loss weight k and p, threshold T Output: Updated model parameters Θ t+1

i,t • L(y i , Φ(x i ; Θ)) -ω i,t • L(g(Φ(x i ; Θ), Φ(x i ; y i , Φ(x i ; Θ)) ∂Θ | T Θ=Θt ∂(L(y i , Φ(x i ; Θ)) -L(g(Φ(x i ; Θ), Φ(x i ; Θ)) ∂Θ | Θ=Θt(11)

Validation accuracy on CIFAR10 with uniform noise. |D p | denotes the number of trusted (probe) data used. 0.1k indicates 10 images per class. For reference, standard training of WRN-28-10/ResNet29 (RN29) leads to 96.1%/92.7% accuracy. * indicates results trained by us.

Table content: <table><tr><td>Method</td><td>|D p |</td><td>0</td><td>0.2</td><td>Noise ratio 0.4</td><td>0.8</td></tr><tr><td>GCE (Zhang &amp; Sabuncu, 2018)</td><td>-</td><td colspan="4">93.5 89.9±0.2 87.1±0.2 67.9±0.6</td></tr><tr><td>MentorNet DD (Jiang et al., 2018)</td><td>5k</td><td>96.0</td><td>92.0</td><td>89.0</td><td>49.0</td></tr><tr><td>RoG (Lee et al., 2019)</td><td>-</td><td>94.2</td><td>87.4</td><td>81.8</td><td>-</td></tr><tr><td>L2R (Ren et al., 2018)</td><td>1k</td><td colspan="4">96.1 90.0±0.4  *  86.9±0.2 73.0±0.8  *</td></tr><tr><td>(Arazo et al., 2019)</td><td>-</td><td>-</td><td>93.8</td><td>92.3</td><td>74.1</td></tr><tr><td>IEG</td><td colspan="5">0.1k 96.8 96.2±0.2 95.9±0.2 93.7±0.5</td></tr><tr><td>IEG-RN29</td><td colspan="4">0.1k 94.4 92.9±0.2 92.5±0.5</td><td>85.6+1.1</td></tr><tr><td>Method</td><td>|D p |</td><td>0</td><td>0.2</td><td>Noise ratio 0.4</td><td>0.8</td></tr><tr><td>GCE (Zhang &amp; Sabuncu, 2018)</td><td>-</td><td colspan="4">81.4 66.8±0.4 61.8±0.2 47.7±0.7</td></tr><tr><td>MentorNet DD (Jiang et al., 2018)</td><td>5k</td><td>79.0</td><td>73.0</td><td>68.0</td><td>35.0</td></tr><tr><td>L2R (Ren et al., 2018)</td><td>1k</td><td colspan="4">81.2 67.1±0.1  *  61.3+2.0 35.1±1.2  *</td></tr><tr><td>(Arazo et al., 2019)</td><td>-</td><td>-</td><td>70.0</td><td>64.4</td><td>45.5</td></tr><tr><td>IEG-RN29</td><td>1k</td><td colspan="4">70.3 69.3±0.5 67.0±0.8 60.7±1.0</td></tr><tr><td>IEG</td><td colspan="5">0.1k 83.0 77.4±0.4 75.1±1.1 62.1±1.2</td></tr><tr><td>IEG</td><td colspan="5">0.5k 83.0 80.4±0.5 79.6±0.3 73.6±1.5</td></tr><tr><td>IEG</td><td>1k</td><td colspan="4">83.0 81.2±0.7 80.2±0.3 75.5±0.2</td></tr></table>

Validation accuracy on CIFAR100 with uniform noise. Standard training of WRN-28-10/RN29 leads to 81.6%/71.3% accuracy. 0.1k indicates 1 images per class.

Table content: <table><tr><td>Method</td><td>0.2</td><td>Noise ratio 0.4</td><td>0.8</td><td>Method RoG</td><td colspan="2">C10 (34%) C100 (37%) 70.0 53.6</td></tr><tr><td>GCE</td><td colspan="2">89.5±0.3 82.3±0.7</td><td>-</td><td>L2R</td><td>71.0  *</td><td>56.9  *</td></tr><tr><td>LC</td><td colspan="2">89.1±0.5 83.6±0.3</td><td>-</td><td>IEG-RN29</td><td>81.8</td><td>65.1</td></tr><tr><td colspan="4">IEG-RN29 92.7±0.2 90.2±0.5 78.9±3.5</td><td>IEG</td><td>88.3</td><td>73.7</td></tr><tr><td>IEG</td><td colspan="3">96.5±0.2 94.9±0.1 79.3±2.4</td><td/><td/></tr><tr><td/><td/><td/><td/><td colspan="3">Table 4: Semantic noisy experiments where</td></tr><tr><td colspan="4">Table 3: Asymmetric noise on CIFAR10. LC is a loss</td><td colspan="3">labels are generated by a neural network on</td></tr><tr><td colspan="4">correction approach (Patrini et al., 2017). 10 trusted</td><td colspan="3">a few data. Noise ratio is shown in paren-</td></tr><tr><td colspan="3">data per class are used as probe data.</td><td/><td colspan="2">theses. RoG uses DenseNet-100.</td></tr></table>

Table 3 compares the results on CIFAR10 with asymmet-Large-scale WebVision experiments. The top-1/top-5 accuracy on the ImageNet validation set are compared. IEG uses ResNet-50. The full version does not use AA.

Table content: <table><tr><td colspan="6">Dataset ResNet-50 (Chen et al., 2019) MentorNet IEG-RN50</td></tr><tr><td>mini</td><td colspan="2">61.0/84.3</td><td colspan="2">61.6/85.0</td><td>63.8/85.8</td><td>72.6/91.5</td></tr><tr><td>full</td><td colspan="2">57.2/79.3</td><td>-</td><td/><td>64.2/84.8</td><td>65.8/85.8</td></tr><tr><td colspan="2">Dataset</td><td colspan="2">MixMatch</td><td>EG</td><td>EG  *</td><td>IEG</td></tr><tr><td colspan="2">CIFAR10</td><td>51.2</td><td colspan="3">92.4±0.7 94.5±0.3 93.7±0.5</td></tr><tr><td colspan="2">CIFAR100</td><td>34.5</td><td colspan="3">57.6±0.4 67.3±0.3 75.2±0.2</td></tr></table>

Comparison with semi-supervised methods. MixMatch and EG use WRN-28-2. EG * and IEG use WRN-28-10. 10 labeled data per class are used. The same size is used for probe data in IEG. The results of IEG are reported under the 80% uniform noise ratio.

Table content: <table/>

.5%) gain when we test baseline ResNet-50 by adding the probe data in training (reported in the table). AA is effective but time consuming. We use standard image augmentation instead of AA for the full Webvision. On mini, standard training ResNet-50 leads to 52.6/84.3 without AA. Standard training IRv2 leads to 57.2/79.2 without AA and 64.0/84.2 with AA.

Table content: <table/>

Table7summarizes the ablation study results (referred as IEG-#) and we discuss them further below. Based on our empirical observations, UC plays an important role in preventing neural networks from overfitting to samples with wrong labels, especially at extreme noise ratios. IEG-4 shows results without UC. FigureA2in Appendix shows the training curves with different coefficient k for L KL . At around 80k iteration, the curve of β = 1 starts to overfit to The effects of input perturbation: IEG-7 shows the results after removing AA-learned policy augmentation (we only use flip → random crop → cutout). Cutout is effective, as also observed in(Xie et al., 2019). Removing it leads to 72.71%/62.41% for 40%/80% noise. Mixup cross-entropy (MC) regularization: Directly minimizing the cross-entropy loss of the tiny probe data would make the model quickly memorize them. IEG-6 shows the result without MC regularization. The performance loss is significant at 80% noise ratio. Therefore, MC effectively brings useful supervision in probe data to guide training. The effects of λ: Our proposed meta re-labeling (equation 3) is very effective for extremely-high noise ratios. It learns to assign lower λ for mislabeled data in order to promote the use of pseudo labels, and vice visa for clean data. Figure2(top) shows the average λ during the training process (the value of noise labels are obtained by peeping ground truth). Figure2(bottom) demonstrates the significant advantage of λ under extreme noise ratios.

Table content: <table><tr><td>Component (abbr.) UC MC AA λ Table 7: Ablation study on CIFAR100. Noise ratio 0.4 0.8 1 64.43 33.52 2 66.14 36.04 3 67.82 37.01 4 78.06 61.81 5 79.96 75.42 6 73.63 54.76 7 79.16 72.69 8 81.05 74.04 / indicates the corresponding component is en-abled/disabled. So IEG-1 is equal to L2R; IEG-5 is the full IEG. Abbreviations are defined in text. noisy labels and simultaneously the validation accuracy starts to decrease. β = 20 is much more 1.0 0 20000 40000 60000 80000 Training steps 0.0 0.2 0.4 0.6 0.8 Average Noisy labels Clean labels 0.80 0.85 0.90 0.93 0.95 0.96 0.98 0.99 Noise ratio 40 50 60 70 Accuracy (%) IEG w/o IEG Figure 2: Analysis of λ. Top: The average λ of noisy and clean labels on CIFAR10 with 40% noise. Bottom: Accuracy (w/o λ) at extreme noise ratios on CIFAR100. Unsupervised consistency (UC): # efficient in overcoming this.</td></tr></table>

Open-set noise on CIFAR10. We follow the setting and created noisy datasets of RoG to conduct experiments. Each column indicates where the noisy out-of-distribution images are from. Three types of noisy types are compared. RoG uses DenseNet-100 and L2R use WRN-28-10. We run the baseline for better comparison (the first block of the table). From results of WRN-28-10, we can see model capacity is beneficial for performance. It is interesting to L2R does not outperforms the its backbone baseline which  implies that only data reweighting is not effective to deal with open-set noise. .85 0.9 0.93 0.95 0.96 0.98 0.99 mean 82.9 81.2 80.2 77.6 75.5 74.7 70.9 68.8 64.8 62.6 58.4 54.4 std 0.25 0.63 0.22 0.35 0.21 0.21 0.45 0.26 0.91 1.85 0.16 0.29

Table content: <table><tr><td/><td colspan="6">Open-set noise type CIFAR100 CIFAR100+ImageNet ImageNet</td></tr><tr><td/><td colspan="2">RN29</td><td/><td/><td>77.8</td><td>80.34</td><td>84.43</td></tr><tr><td/><td colspan="3">DenseNet-100</td><td/><td>79.0</td><td>86.7</td><td>81.6</td></tr><tr><td/><td colspan="2">WRN-28-10</td><td/><td/><td>82.8</td><td>84.7</td><td>88.7</td></tr><tr><td/><td>L2R</td><td/><td/><td/><td>81.8</td><td>81.3</td><td>85.0</td></tr><tr><td/><td>RoG</td><td/><td/><td/><td>83.4</td><td>87.1</td><td>84.4</td></tr><tr><td/><td colspan="2">IEG-RN29</td><td/><td/><td>86.4</td><td>87.4</td><td>90.0</td></tr><tr><td/><td>IEG</td><td/><td/><td/><td>92.3</td><td>93.0</td><td>94.0</td></tr><tr><td>Ratio</td><td>0</td><td>0.2</td><td>0.4</td><td>0.6</td><td>0.8 0</td></tr></table>

Accuracy (mean and std) of IEG on CIFAR100 with different uniform noise ratios.

Table content: <table><tr><td>1 M</td><td>M i=1</td><td>∂ ∂λ i,t</td><td>L</td></tr></table>

Peer review (iclr2020):
Summary: The paper proposed a IEG method to handle the noisy label learning problem. IEG consists of 3 parts: 1) isolate the noise labels via meta optimization; 2) escalate the supervision mislabeled data via pseudo meta re-labeling; 3) use small trusted data to guide training. IEG gains improvements over SOTA. Strength: 1. The motivation of IEG is very clear. 2. Ablation experiments are useful for understanding. Weakness: 1. The organization of Section 2 is somewhat confusing. Readers may need to jump from one section to another section to follow how the method work. IEG should consist of 3 components, while there are only 2 subsections in section 2. Could the authors clarify which part corresponds to which component? 2. I am wondering where are the tiny trusted data used? I am not following that from ALG 1. Do I miss something? 3. For experimental comparisons, I think the authors may miss some important baselines. 1) Symmetric cross entropy for robust learning with noisy labels, ICCV2019 2) Joint Optimization Framework for Learning with Noisy Labels, CVPR2018 3) Dimensionality-driven learning with noisy labels, ICML2018 4. For the ablation study, the authors miss some important experiments. 1) testing the influence of the number of trusted data; 2) showing the rate of isolated misclassified data in the learning process. 3) showing the influence of weight w. 5. Open-set case is important for understanding IEG. I am confusing why IEG still works so well in the open-set case where most labels of data cannot be predicted correctly because of the open-set setting. The weight w before examples very small? or anything else? The author should provide more evidence on open-set case for better understanding.