Title: META LABEL CORRECTION FOR LEARNING WITH WEAK SUPERVISION

Abstract: Leveraging weak or noisy supervision for building effective machine learning models has long been an important research problem. The growing need for largescale datasets to train deep learning models has increased its importance. Weak or noisy supervision could originate from multiple sources including non-expert annotators or automatic labeling based on heuristics or user interaction signals. Previous work on modeling and correcting weak labels have been focused on various aspects, including loss correction, training instance re-weighting, etc. In this paper, we approach this problem from a novel perspective based on meta-learning. We view the label correction procedure as a meta-process and propose a new meta-learning based framework termed MLC for learning with weak supervision. Experiments with different label noise levels on multiple datasets show that MLC can achieve large improvement over previous methods incorporating weak labels for learning.

1. INTRODUCTION:

Recent advances in deep learning have enabled several natural language processing models to achieve impressive performance. At the core of this success lies the availability of large amounts of annotated data. However, such datasets are not readily available in large scale for many tasks. The problem of learning with weak supervision aims to address this challenge by leveraging weak evidence for supervision, such as corrupted labels, noisy labels,automatic labels based on heuristics or user interaction signals, etc. Two major lines of work have been proposed to combine trusted (or gold) labeled data with weak or noisy supervision data for better learning. The first approach relies on re-weighting of training instances (Ren et al., 2018) . It aims to assign proper importance to each sample in the training set such that the ones with higher weights will contribute more positively to the model training. On the other hand, the second approach relies on the idea of label correction. It aims to correct the noisy/corrupt labels based on certain assumptions about the weak label generation process. In a sense, label correction is a finer way to incorporate the noisy data samples than simply assigning scalar weights to each training instance and has shown to work well even in the setting where a very small set of clean labels is available. Label correction in previous methods relies on the assumption about the weak label generation process and thus often involves estimating a label corruption matrix (Hendrycks & Gimpel, 2016) . However, the label correction estimation is done in separation from the main model limiting the flow of information between them.

In this paper, we address the label correction problem from a novel angle based on meta-learning and propose meta label correction (MLC). Specifically, we view the label correction procedure as a meta-process, meaning that its objective is to provide corrected labels for the examples with weak labels. On the other hand, the main supervised model is trained to fit the corrected labels (generated by the meta-model). Both the meta-model and the main model are learned by optimizing the model performances on the gold data set (i.e., a validation set w.r.t. the noisy set) allowing us to co-optimize the label correction process and the main model process.

Meta learning has been successfully used for many applications including hyper-parameter tuning (Maclaurin et al., 2015) , model selection (Pedregosa, 2016) and neural architecture search (Liu et al., 2018) . To the best of our knowledge, MLC is the first to utilize a meta model to automatically "tune" noisy labels from data and combine it with trusted labels for better learning. The contributions of this paper can be summarized as follows:

• A new learning framework with weak supervision based on meta learning, is proposed based on a novel angle by treating the label correction network as a meta process to provide reliable labels for the main models to learn;

• We conduct experiments on various text classification and gray-scale image recognition experiments and the proposed methods outperform previous best methods on label correction, demonstrating the power of the proposed method.

The rest of the paper is organized as follows: We briefly review the preliminaries on learning with weak supervision, particularly on learning with corrupt/noisy labels 2 and propose a meta-learning based learning framework for weak supervision in Section 3. Empirical evaluations and analysis are conducted in Section 4 and we conclude the paper in Section 6.

2. PRELIMINARIES:

We follow a setup of learning with weak supervision that involves two sets of data: a small set of data with clean/reliable labels {x i , y i } m i=1 and a large set of data with weak supervision (noisy/corrupted labels) {x i , y i } M i=1 . Typically the clean set is much smaller compared to the noisy set (m M ) due to scarcity of expert labels and to labeling costs. Training directly on the small clean set often tends to be sub-optimal, as too little data can easily cause over-fitting. The problems of learning with weak supervision under this setup can then be formulated as how to build a predictive model f : X → Y with the given two sets. Two major lines of work have been proposed to solve this problem.

2.1. LEARNING WITH LABEL CORRECTION:

The first line of work aims to correct the weak labels as much as possible by imposing assumptions of how the noisy labels are generated from its underlying true labels. To be concrete, consider the problem of classifying the data into k categories, where label correction involves estimating a label corruption matrix C k×k whose entry C ij denotes the probability of observing weak label for class i while the underlying true class label is actually j. Gold loss correction (Hendrycks et al., 2018) falls into this category; a key drawback of this line of work is that the label perturbing matrix is often estimated in an ad-hoc way and also that the estimation process is separate from the main model process, hence allowing no feedback from the main model to the estimation process.

2.2. LEARNING TO RE-WEIGHT TRAINING INSTANCES:

Knowing that not all training examples are equally important and useful for building a main model, another line of work for learning with weak supervision is to assign learnable weights to each example in the training noisy set. The goal is to assign a a proper weight for each training example such that the main model would perform well on a separate validation set (the clean set) (Ren et al., 2018; ?) . The example weights are essentially hyper-parameters for the main model and can be learned by formulating a bi-level optimization problem. Due to the meta-learning characteristic of this framework, the example weights learning and the main model could communicate with each other and a better model could be learned.

3. META LABEL CORRECTION:

One advantage of the label correction approach is that it allows us to combine trusted labels and corrected noisy labels in the learning process. Our proposed approach adopts the label correction methodology while also co-optimizing the label correction process together with the main model process through an unified meta learning framework. We achieve that by adopting a meta-learning framework where the meta learner (meta model) tries to correct the noisy labels and the main model tries to build the best predictive model with corrected labels coming from the meta model, allowing the meta model and main model to reinforce each other. The meta model g α (in orange) takes in a noisy example label pair and tries to generate a "corrected" label which will be treated as the correct labels to train the main model f w (in blue). Thus the trained main model depends on the corrected labels y c , hence further depends on the parameters of the meta model. The trained main model then will be tested on a separate true clean set, whose loss is to be minimized. Note that when minimizing the loss on the clean examples, the parameters for the main models are not changed, only to let the loss signal on the clean examples to propogate back to the meta network, thus making g α generate better corrections for x, y . In practice, we won't be able to always get the trained main model to evaluate on the clean examples, thus k-step SGD ahead updated version of the main model is used as a "trained" model.

We describe the framework in detail as follows. Given a set of clean data examples D = {x, y} m and a set of noisy data examples D = {x, y } M with m much smaller than M . To best utilize the information provided by the weak labels, we propose to construct a label correction network (LCN), serving as a meta model, which takes a pair of noisy data example and its weak label as input and produces a different version of the weak label. Formally, the label correction network (LCN) is defined as a function with parameters α:

EQUATION

to correct the weak label y of example x to its true label. (Note that the subscription in y c emphasizes that it's generating a corrected label). Meanwhile, the main model f , that we aim to train and use for prediction after training, is instantiated as another function with parameters w,

EQUATION

Without linking the two models, there's no way to enforce that 1.) the generated label for an example from the meta model g is indeed a meaningful one, let alone a corrected one, since we cannot directly train the meta model without clean labels for the noisy examples ; 2) The main model f might be fitting onto arbitrary targets, if the labels provided do not align with the unknown true labels. Fortunately, the two models can be linked together via a bi-level optimization problem, by the intuition that if the labels generated by the meta model is of high quality, then we can use these pairs of examples and their corrected labels as training data to train a good main model, such that the main model achieves low loss on a separate set of clean examples. This can be instantiated as the following bi-level optimization problem:

EQUATION

where () denotes a chosen differentiable loss function to measure the predictive error and the subscript of w is to emphasize the dependency of the best main model f on α. We term this framework as Meta Label Correction (MLC); Figure 1 provides an overview of the framework.

In this bi-level formulation, since the LCN is parameterized by α, α are the upper parameters (or meta parameters) while the main model parameters w are the lower parameters (or main parameters). Like many other work involving bi-level optimizations, exact solutions of Problem (3) requires solving for the optimal w * whenever α is updated. This is often analytically infeasible when the while not converged do Update meta parameters α by descending ∇ α L D (w -kη∇ w L D (α, w)) Update model parameters w by descending ∇ w L D (α, w) end Algorithm 1: MLC -Meta Label Correction main model f is complex, such as deep neural networks, and also computationally expensive. Instead of solving for the optimal for w * for each α, we use a k-step look ahead SGD update for w as an estimate to the optimal main model for a given α

EQUATION

where L D (α, w) = E (x,y )∈D (g α (x, y ), f w (x)), thus the proxy optimization problem turns to

EQUATION

Algorithm 1 outlines an iterative procedure to solve the above proxy problem with k-step look ahead SGD for the main model:

The above meta learning algorithm involves computing an expensive second-order partial derivative ∇ 2 α,w L D (α, w) followed by a matrix vector product. To speed up training, we propose to approximate the second order gradients with finite differences, as follows

EQUATION

where w ± = w ± ∇ w L D (w ), and w = w -kη∇ w L D (α, w). Similar approximation strategy is also adopted by meta-leanring for architecture search. (Liu et al., 2018; Finn et al., 2017) 

3.1. CONVERTING A CLASSIFIER TO A LABEL CORRECTION NETWORK:

There are multiple ways to build mappings from (x, y ) with deep neural networks as the desired label correction network. In essence, g α (x, y ) behaves also like a classifier with the only difference from conventional classifiers, i.e., it also takes the noisy label y as input. To ease the effort of designing and working with the MLC framework, we explored several simple strategies, which doesn't require heavy-weight modifications to existing architectures. The one that is adopted in all our experiments is by constructing the LCN as a weighted combination from a classifier h(x) and the weak label y itself, i.e.

g(x, y ) ≡ λ(x)h(x) + (1 -λ(x))y (9)

where λ is a data dependent scalar controlling the mixing weights. We found it helps to have a separate λ for each class, hence different weak classes fed in will be paired with different weights.

If doing so, this only requires modifying the last layer of the classifier h(x), to output a vector of dimension 2C (C dimensions for the class logits, and the rest C dimensions for the weak label weights λ), instead of C for the original classifier (where C is number of classes),

3.2. SOFT CROSS ENTROPY LOSS FOR LEARNING WITH WEAK SUPERVISION:

In the classification scenario, when a clean label is given to a data example typically the cross entropy loss is used to train the classifier. Here, we demonstrate that with a soft label (generated from the label correction network), how the soft cross entropy loss could be beneficial for the weakly supervised setting. Denote the the soft label as q, where q is a dense vector with i q i = 1, typically resulted from a softmax layer and denote the predicting probability of the main classifier as p with i p i = 1.

In this setup, the original cross entropy loss defined for hard labels can be naively extended as

CE soft (p, q) = - i q i log p i = i q i log q i p i - i q i log q i = KL(q, p) + entropy(q) (10)

Minimizing this loss w.r.t the parameters of the main model is equivalent to (with the meta model fixed, thus q fixed) minimizing the KL divergence between the (soft) label and the predicting distribution, similar to the hard label case. And when updating the parameters of the meta model, minimizing this loss function is now equivalent to (with the main model fixed, thus p fixed) minimizing both the KL divergence between the (soft) label and the predicting distribution, and also the entropy of the soft labels predicted by the meta model, since we would like to have labeling distribution as close to discrete as possible.

3.3. k-STEP LOOK AHEAD SGD LOOK AHEAD IN META MODEL LEARNING:

We found it's crucial to use a value of k greater than 1 for MLC to ensure model convergence, particularly in the early phases of training, when both the main model and the meta model are close to random predictors and lacks confidence in their outputs. Using k = 1 is likely to confuse both models and thus they won't converge. This is not the case for previous similar works with meta-learning, however we find this to be crucial, as the main model in GLC is not directly trained on any clean examples, thus slightly more explorations from the main model is likely to help training convergence. We'll explore this aspect in the coming section. Due to this requirement, in all our experiments, we use scheduling for k starting from 1500 and decreasing to 500 towards the end of model training.

4. EXPERIMENTS:

To test the performance of MLC, we conduct experiments on a set of classification tasks, both from the text and vision domains, and compare results with previous state-of-the-art approaches for learning with weak/noisy labels, under different weak label scenarios.

4.1. WEAK SUPERVISION GENERATION:

To generate weak supervision data, for each data set we test on, we sample a portion of the entire training set as the clean set. The noisy dataset is generated by corrupting the labels of all the remaining data points based on one of the following two(three) settings:

• Uniform mixture (UNIF)

• Flipped labels (FLIP)

• Weak labels from trained weak classifiers (WEAK)

The first two methods follow the same procedure adopted by (Hendrycks et al., 2018) by either corrupting uniformly all classes or by flipping a label to a different class. To generate the corrupted labels from the true labels, we first devise a corruption probability categorical distribution for each true class, hence for all classes the corruption probability forms a label corruption matrix C. Then, for an example with true label i, we sample the corrupted label from the categorical distribution parameterized by the ith row of C. Note that this is a simplified assumption assuming that the corrupted label does not depend on the data example itself, however we still use this to ensure a fair comparison to (Hendrycks et al., 2018) where the same process was used for generating noisy data.To create a noisy datasets with different levels of noise, we take a convex combination of an identity matrix and the corruption matrix, with the coefficient of the latter serving as an indicator of the noise level (Hendrycks et al., 2018) .

To also study scenarios where the noise could be dependent on both the data and the label, we introduce a third more realistic method: WEAK. In this method, weak labels are provided by separate (weak) predictive models that depend on both the data and the labels. To generate different levels of noise, we train multiple weak predictive models with varying accuracies where a lower accuracy corresponds to a higher noise level. Note that in all noise levels, the weak predictive model is performing better than random prediction.

Note that all method are not aware of this true label corruption probability nor do they have knowledge about which data sample in the noisy set is actually corrupted. Since UNIF and FLIP are similar to one another, we report results based on UNIF only and leave the FLIP results for the appendix.

4.2. BASELINE METHODS:

We test MLC mainly against the current state-of-the-art model (Hendrycks et al., 2018) for label correction (denoted by GLC hereafter) in various settings where the labels in the noisy set are corrupted by different noise levels, as well as the different ratios the clean set is sampled from the entire training set. Note that GLC was shown to perform consistently better than other models that combine the clean and weak labels using methods such as distillation Li et al. (2017) . For completeness, we also compare with the forward loss correction method proposed in (Sukhbaatar et al., 2014) (denoted by Forward hereafter). In lieu with meta learning for learning with weak supervision, we also compare with the method of learning weights of training examples to learn a robust classifier (Ren et al., 2018) (denoted by L2R hereafter).

4.3. DATA SETS AND IMPLEMENTATION DETAILS:

To ensure fair comparison with previous methods as much as possible, we experiment on a broad set of data collections from both , with our best effort to match the pre-processing and hyper-parameter setting from previous methods when experimenting with them on new datasets that were not used in the original papers.

We test on 10 different collections with varying data sizes. To compare with GLC, we test on all three text collections used by (Hendrycks et al., 2018) and on the MNIST dataset. The dataset are::

MNIST: The MNIST dataset contains 28 × 28 grayscale images of the digits 0-9. The training set has 60,000 images and the test set has 10,000 images. For preprocessing, we rescale the pixels to the interval [0, 1]. We train a 2-layer fully connected network with 128 hidden dimensions. We train with Adam for 20000 iterations using batches of size 128 and a learning rate of 0.001 for the main model and 0.0001 for the meta model.

Twitter: The Twitter Part of Speech dataset (Gimpel et al., 2011) contains 1,827 tweets annotated with 25 POS tags. This is split into a training set of 1,000 tweets, a development set of 327 tweets, and a test set of 500 tweets. We use the development set to augment the training set. We use the same pretrained 50-dimensional word vectors as in (Hendrycks et al., 2018) , and for each token, we concatenate word vectors in a fixed window centered on the token. These form our training and test set. We use a window size of 3, and train a 2-layer fully connected network with hidden size 256, and use the GELU nonlinearity (Hendrycks & Gimpel, 2016) . We train with Adam (Kingma & Ba, 2014) for 20000 iterations with batch size 128 and learning rate of 0.001 for the main model and 0.0001 for the meta model. Wwe use 2 weight decay with λ = 3 × 10 -4 on all the weights.

SST::

The Stanford Sentiment Treebank dataset consists of single sentence movie reviews (Socher et al., 2013) . We use the 2-class version (i.e. SST2), which has 6,911 reviews in the training set, 872 in the development set, and 1,821 in the test set. We follow the same data and model setups as in g (Hendrycks et al., 2018) ; the classifier is a word-averaging model with an affine output layer. We use the Adam optimizer for 10000 epochs with batch size 50 and learning rate 0.001. For regularization, we use 2 weight decay with λ = 1 × 10 -4 on the output layer.

IMDB::

The IMDB datasets contains 50k movie reviews from IMDB, with 25k positive and 25k negative. We use a one-layer LSTM (Hochreiter & Schmidhuber, 1997) for both main model and the meta model..

The above datasets are relatively small text collections, though meaningful to demonstrate the different methods for learning with weak supervision with simple classifier architectures. We also include a range of 6 large scale text classification benchmark datasets including:

AG News: AG is a text classification dataset derived from a large collection of news articles gathered from more than 2000 news sources. Each news article is categorized to 1 of 4 classes, including World, Sports, Business and Sci/Tech.1 . This dataset has 120,000 training examples and 7,600 examples for testing.

Amazon Reviews (Amazon-2 and Amazon-5): The Amazon-2 and Amazon-5 datasets contain randomly sampled customer reviews from Amazon. The Amazon-2 is a binary polarity rating classification dataset while Amazon-5 is a rating classification dataset on a scale from 1 to 5. The Amazon-2 dataset has 3,600,000 training samples and 400,000 testing samples, while the Amazon-5 dataset has 3,000,000 trainig samples and 650,000 testing samples.

Yelp Reviews (Yelp-2 and Yelp-5): The Yelp-2 and Yelp-5 datasets are constructed from the Yelp Dataset Challenge 2015 data, for binary polarity rating classification and 5-way rating classification, respectively. Yelp-2 contains 560,000 training samples and 38,000 testing samples and Yelp-5 contains in total of 650,000 training samples and 50,000 testing samples.

DBpedia DBpedia is a crowd-sourced project aiming to extract structured information from Wikipedia. The DBpedia dataset was covers 14 non-overlapping ontology classes from DBPedia. Each class contains 40,000 training samples and 5,000 testing samples. Hence, the full dataset has 560,000 training samples and 70,000 testing samples.

For all the large scale text classification datasets (AG, Amazon-2 and -5, Yelp-2 and 5 and DBpedia), we adopt a pre-trained BERT-base (Devlin et al., 2018) model for both the main network and meta network. This ensures that we can test the ability of MLC in the weakly supervised setting with strong state-of-the-art base models.

We implement all models and experiments in PyTorch2 . To ensure fair comparison, we adopt the same main network architecture as much as possible from previous best methods with comparable number of parameters. A brief overview of the neural net architectures used in various settings is listed in Table 1 (Refer to the appendix for a detailed description of the model architectures). Code for reproducing the results in this paper will be made publicly available.

4.4. MAIN RESULTS:



MLC with MLP, LSTM::

We investigate multiple settings with an extensive set of different configurations, i.e., two noise types, different noise levels, and different clean ratios. Table 2 presents the averaged accuracies across all these configurations with each one repeated for 5 times. Notice that the results vary per dataset when the news is generated using UNIF (i.e. noise is independent of the data).

On the other hand, we notice that the performance of all methods seems to drop when we use WEAK (noise depends on the data and the label). This shows that this is a more realistic and challenging settings. We also observe that MLC performs consistently better in this case. 3 presents the error rates of MLC on 6 large text data sets with pretrained BERT-base as its main model and meta models. Note that these are much larger scale dataset and that the baselines and the base models for both the meta and the main learner are using Bert. We notice that MLC consistently outperforms the baselines (except for L2R on the AG dataset). 

4.5. DETAILED RESULTS:

We investigate how the noise levels in the weak labels affect MLC training. Due to space limitations, we only present detailed results on Twitte and MNIST. Detailed experiments on SST and IMDB can be found in the appendix.

Twitter. Figure 2 presents the detailed performances with different clean data ratio and label noise levels. For the UNIF setting, both loss correction methods (GLC and MLC) works better than using only clean data to train the classifier, emphasizing the importance of incorporating those weakly supervised examples. With 1% and 5% only clean data, MLC achieves consistently higher accuracies over the range of high noise levels, implying the robustness of MLC with severe noise present. In the WEAK setting, where the weak labels are generated by weak classifiers, GLC performs worse than MLC since it assumes that the noisy labels are only dependent on the true label but not on the data. In contrast, MLC gains significant edge over the other methods as MLC doesn't make such assumptions. MNIST. Figure 3 presents the detailed performances with different gold data ratio and corruption levels. Similar trends could be observed as previously seen in Twitter and SST. On the UNIF setting, MLC is not as good as GLC and L2R; however in the upper range of noise levels, MLC catches up the MAE of the estimated label corruption matrix, we verify that the corrected label distribution aligns well to the ground truth for the UNIF setting (so is GLC, however GLC cannot be used to correct future unseen examples).

5. RELATED WORK:

Labeled data largely determines whether a machine learning system can perform well on a task or not, as noisy label or corrupted labels could cause dramatic performance drop (Nettleton et al., 2010) . The problem gets even worse when an adversarial rival intentionally injects noises into the labels (Reed et al., 2014) . Thus, understanding, modeling, correcting, and learning with noisy labels has been of interest at large in the research communities (Natarajan et al., 2013; Frénay & Verleysen, 2013) .

Several works (Mnih & Hinton, 2012; Patrini et al., 2017; Sukhbaatar et al., 2014; Larsen et al., 1998) have attempted to address the weak labels by modifying the model's architecture or by implementing a loss correction. (Sukhbaatar et al., 2014) introduced a stochastic variant to estimate label corruption, however the methods have to have access to the true labels, rendering it inapplicable when no true labels are present. A forward loss correction adds a linear layer to the end of the model and the loss is adjusted accordingly to incorporate learning about the label noise. (Patrini et al., 2017 ) also make use of the forward loss correction mechanism, and propose an estimate of the label corruption estimation matrix which relies on strong assumptions, and does not make use of clean labels that might be available for a portion of the data set.. Following (Charikar et al., 2017) , we assume that during training the model has access to a small set of clean labels besides a large set of weak labels. This assumption has been leveraged by others for the purpose of label noise robustness, most notably (Veit et al., 2017; Li et al., 2017; Xiao et al., 2015; Ren et al., 2018) . (Veit et al., 2017) use human-verified labels to train a label cleaning network by estimating the discrepancies between the noisy and clean labels in a multi-label classification setting. This assumes that, for a subset of the data, both trusted and noisy labels are available. This work avoids this limitation by proposing a meta learning approach that does not require trusted and noisy data to be available for the same instances.

6. CONCLUSIONS:

In this paper, we address the problem of learning with weak supervision from a meta-learning perspective. Specifically, we propose to use a meta network to correct the noisy labels from the noisy data set, and a main classifier network is trained to fit the example to a provided label, i.e., corrected labels for the noisy examples and true labels for the clean examples. The meta network and main network are jointly optimized in a bi-level optimization framework; to address the computation challenge, we employ a k-step ahead SGD update for the model weights of the main model. Empirical experiments on several benchmark datasets including text and graysacle images demonstrates the effectiveness of MLC.

A ADDITIONAL RESULTS

A.1 SST AND IMDB:

SST. Figure 6 presents the detailed performances with different gold data ratio and corruption levels. On this binary classification task, it's surprising to observe that using clean data solely is only achieving results that are slightly better than random guessing (an accuracy of 0.5). Again with the help of label correction for the noisy examples, the performance boosts by quite a margin with GLC. So does MLC over GLC, demonstrating the potential power of using a meta network as a label correction procedure. All numbers reported are accuracies on the test set. For references, using gold data only to train a model yields test accurucies of (0.541?, 0.647? and 0.741?, for three gold data ratios respectively. 5) 0.867 ± 0.000(5) 0.852 ± 0.000(1) 0.1 0.534 ± 0.088(5) 0.868 ± 0.003(5) 0.861 ± 0.000(5) 0.829 ± 0.000(1) 0.2 0.494 ± 0.087(5) 0.865 ± 0.002(5) 0.854 ± 0.000(5) 0.824 ± 0.000(1) 0.3 0.499 ± 0.112(5) 0.865 ± 0.006(5) 0.848 ± 0.000(5) 0.818 ± 0.000(1) 0.4 0.468 ± 0.064(5) 0.861 ± 0.008(5) 0.837 ± 0.000(5) 0.812 ± 0.000(1) 0.5 0.348 ± 0.066(5) 0.859 ± 0.014(5) 0.819 ± 0.000(5) 0.797 ± 0.000(1) 0.6 0.341 ± 0.020(5) 0.857 ± 0.007(5) 0.800 ± 0.000(5) 0.801 ± 0.000(1) 0.7 0.272 ± 0.054(5) 0.856 ± 0.005(5) 0.774 ± 0.000(5) 0.791 ± 0.000(1) 0.8 0.315 ± 0.037(5) 0.845 ± 0.025(5) 0.724 ± 0.000(5) 0.786 ± 0.000(1) 0.9 0.326 ± 0.106(5) 0.820 ± 0.038(5) 0.579 ± 0.000(5) 0.781 ± 0.000(1) 1.0 0.190 ± 0.080(5) 0.800 ± 0.032(5) 0.220 ± 0.000(5) 0.778 ± 0.000(1) Mean 0.393 ± 0.026(5) 0.851 ± 0.007(5) 0.744 ± 0.000(5) 0.806 ± 0.000(1)

Clean ratios 10.0%

Noise Level Forward GLC L2R MLC 0.0 -0.866 ± 0.012(5) 0.866 ± 0.002(5) 0.875 ± 0.000(1) 0.1 -0.866 ± 0.003(5) 0.855 ± 0.000(5) 0.858 ± 0.000(1) 0.2 -0.863 ± 0.004(5) 0.847 ± 0.000(5) 0.850 ± 0.000(1) 0.3 -0.864 ± 0.002(5) 0.843 ± 0.000(5) 0.843 ± 0.000(1) 0.4 -0.865 ± 0.006(5) 0.836 ± 0.000(5) 0.835 ± 0.000(1) 0.5 -0.863 ± 0.005(5) 0.819 ± 0.000(5) 0.833 ± 0.000(1) 0.6 -0.863 ± 0.007(5) 0.795 ± 0.000(5) 0.826 ± 0.000(1) 0.7 -0.855 ± 0.014(5) 0.763 ± 0.000(5) 0.822 ± 0.000(1) 0.8 -0.861 ± 0.006(5) 0.724 ± 0.000(5) 0.822 ± 0.000(1) 0.9 -0.858 ± 0.009(5) 0.572 ± 0.000(5) 0.816 ± 0.000(1) 1.0 -0.851 ± 0.010(5) 0.201 ± 0.000(5) 0.805 ± 0.000(1) Mean -0.861 ± 0.003(5) 0.738 ± 0.000(5) 0.835 ± 0.000(1) 5) 0.257 ± 0.082(5) 0.554 ± 0.005(5) 0.616 ± 0.000(1) 2 0.189 ± 0.062(5) 0.173 ± 0.040(5) 0.403 ± 0.013(5) 0.623 ± 0.000(1) 3 0.131 ± 0.021(5) 0.101 ± 0.031(5) 0.350 ± 0.011(5) 0.645 ± 0.000(1) Mean 0.194 ± 0.019(5) 0.177 ± 0.020(5) 0.436 ± 0.006(5) 0.628 ± 0.000(1)

Clean ratios 5.0%

Weak Classifier Forward GLC L2R MLC 1 0.237 ± 0.046(5) 0.339 ± 0.061(5) 0.555 ± 0.006(5) 0.760 ± 0.000(1) 2 0.230 ± 0.061(5) 0.302 ± 0.057(5) 0.400 ± 0.008(5) 0.759 ± 0.000(1) 3 0.181 ± 0.021(5) 0.238 ± 0.023(5) 0.358 ± 0.007(5) 0.773 ± 0.000(1) Mean 0.216 ± 0.022(5) 0.293 ± 0.032(5) 0.438 ± 0.006(5) 0.764 ± 0.000(1)

Clean ratios 10.0%

Weak Classifier Forward GLC L2R MLC 1 0.291 ± 0.037(5) 0.485 ± 0.043(5) 0.549 ± 0.005(5) 0.790 ± 0.000(1) 2 0.296 ± 0.057(5) 0.422 ± 0.078(5) 0.397 ± 0.011(5) 0.789 ± 0.000(1) 3 0.218 ± 0.030(5) 0.338 ± 0.039(5) 0.349 ± 0.021(5) 0.802 ± 0.000(1) Mean 0.268 ± 0.012(5) 0.415 ± 0.043(5) 0.432 ± 0.008(5) 0.794 ± 0.000(1) 0.0 0.810 ± 0.002(5) 0.812 ± 0.001(5) 0.747 ± 0.000(5) 0.646 ± 0.000(1) 0.1 0.804 ± 0.005(5) 0.813 ± 0.004(5) 0.715 ± 0.006(5) 0.625 ± 0.000(1) 0.2 0.802 ± 0.006(5) 0.812 ± 0.005(5) 0.685 ± 0.000(5) 0.610 ± 0.000(1) 0.3 0.795 ± 0.005(5) 0.808 ± 0.002(5) 0.666 ± 0.000(5) 0.593 ± 0.000(1) 0.4 0.789 ± 0.003(5) 0.803 ± 0.005(5) 0.640 ± 0.000(5) 0.578 ± 0.000(1) 0.5 0.781 ± 0.004(5) 0.796 ± 0.005(5) 0.595 ± 0.000(5) 0.545 ± 0.000(1) 0.6 0.757 ± 0.003(5) 0.777 ± 0.007(5) 0.536 ± 0.004(5) 0.535 ± 0.000(1) 0.7 0.734 ± 0.004(5) 0.750 ± 0.018(5) 0.524 ± 0.000(5) 0.520 ± 0.000(1) 0.8 0.685 ± 0.007(5) 0.675 ± 0.087(5) 0.510 ± 0.000(5) 0.526 ± 0.000(1) 0.9 0.618 ± 0.009(5) 0.525 ± 0.045(5) 0.506 ± 0.000(5) 0.517 ± 0.000(1) 1.0 0.505 ± 0.009(5) 0.501 ± 0.003(5) 0.504 ± 0.000(5) 0.512 ± 0.000(1) Mean 0.735 ± 0.003(5) 0.734 ± 0.011(5) 0.603 ± 0.001(5) 0.564 ± 0.000(1)

Clean ratios 1.0%

Noise Level Forward GLC L2R MLC 0.0 0.807 ± 0.003(5) 0.810 ± 0.003(5) 0.797 ± 0.000(5) 0.712 ± 0.000(1) 0.1 0.800 ± 0.005(5) 0.810 ± 0.004(5) 0.781 ± 0.000(5) 0.693 ± 0.000(1) 0.2 0.796 ± 0.005(5) 0.810 ± 0.005(5) 0.769 ± 0.000(5) 0.689 ± 0.000(1) 0.3 0.794 ± 0.006(5) 0.807 ± 0.003(5) 0.763 ± 0.000(5) 0.671 ± 0.000(1) 0.4 0.790 ± 0.005(5) 0.801 ± 0.007(5) 0.726 ± 0.000(5) 0.662 ± 0.000(1) 0.5 0.779 ± 0.004(5) 0.794 ± 0.007(5) 0.667 ± 0.000(5) 0.655 ± 0.000(1) 0.6 0.759 ± 0.005(5) 0.781 ± 0.009(5) 0.542 ± 0.000(5) 0.643 ± 0.000(1) 0.7 0.738 ± 0.004(5) 0.734 ± 0.049(5) 0.523 ± 0.000(5) 0.608 ± 0.000(1) 0.8 0.694 ± 0.006(5) 0.593 ± 0.110(5) 0.501 ± 0.000(5) 0.605 ± 0.000(1) 0.9 0.628 ± 0.010(5) 0.576 ± 0.085(5) 0.501 ± 0.000(5) 0.568 ± 0.000(1) 1.0 0.519 ± 0.014(5) 0.519 ± 0.026(5) 0.501 ± 0.000(5) 0.567 ± 0.000(1) Mean 0.737 ± 0.003(5) 0.731 ± 0.020(5) 0.643 ± 0.000(5) 0.643 ± 0.000(1)

Clean ratios 5.0%

Noise Level Forward GLC L2R MLC 0.0 0.810 ± 0.003(5) 0.810 ± 0.002(5) 0.761 ± 0.000(5) 0.769 ± 0.000(1) 0.1 0.805 ± 0.006(5) 0.812 ± 0.002(5) 0.734 ± 0.000(5) 0.768 ± 0.000(1) 0.2 0.801 ± 0.008(5) 0.813 ± 0.004(5) 0.756 ± 0.000(5) 0.766 ± 0.000(1) 0.3 0.797 ± 0.006(5) 0.812 ± 0.006(5) 0.675 ± 0.000(5) 0.757 ± 0.000(1) 0.4 0.789 ± 0.007(5) 0.806 ± 0.004(5) 0.607 ± 0.000(5) 0.751 ± 0.000(1) 0.5 0.780 ± 0.007(5) 0.804 ± 0.006(5) 0.509 ± 0.000(5) 0.744 ± 0.000(1) 0.6 0.761 ± 0.007(5) 0.788 ± 0.009(5) 0.506 ± 0.000(5) 0.736 ± 0.000(1) 0.7 0.736 ± 0.014(5) 0.720 ± 0.111(5) 0.500 ± 0.000(5) 0.728 ± 0.000(1) 0.8 0.701 ± 0.011(5) 0.661 ± 0.124(5) 0.499 ± 0.000(5) 0.687 ± 0.000(1) 0.9 0.646 ± 0.008(5) 0.589 ± 0.109(5) 0.499 ± 0.000(5) 0.675 ± 0.000(1) 1.0 0.559 ± 0.009(5) 0.572 ± 0.071(5) 0.499 ± 0.000(5) 0.669 ± 0.000(1) Mean 0.744 ± 0.005(5) 0.744 ± 0.030(5) 0.595 ± 0.000(5) 0.732 ± 0.000(1) Clean ratios 1.0%

Noise Level Forward GLC L2R MLC 0.0 0.807 ± 0.003(5) 0.810 ± 0.003(5) 0.797 ± 0.000(5) 0.731 ± 0.000(1) 0.1 0.796 ± 0.005(5) 0.810 ± 0.005(5) 0.758 ± 0.000(5) 0.710 ± 0.000(1) 0.2 0.790 ± 0.005(5) 0.801 ± 0.007(5) 0.726 ± 0.000(5) 0.690 ± 0.000(1) 0.3 0.759 ± 0.005(5) 0.781 ± 0.009(5) 0.624 ± 0.000(5) 0.664 ± 0.000(1) 0.4 0.694 ± 0.006(5) 0.593 ± 0.110(5) 0.501 ± 0.000(5) 0.634 ± 0.000(1) 0.5 0.519 ± 0.014(5) 0.519 ± 0.026(5) 0.501 ± 0.000(5) 0.584 ± 0.000(1) 0.6 0.323 ± 0.008(5) 0.641 ± 0.110(5) 0.499 ± 0.000(5) 0.566 ± 0.000(1) 0.7 0.251 ± 0.010(5) 0.773 ± 0.010(5) 0.498 ± 0.000(5) 0.551 ± 0.000(1) 0.8 0.227 ± 0.009(5) 0.795 ± 0.006(5) 0.499 ± 0.000(5) 0.535 ± 0.000(1) 0.9 0.204 ± 0.008(5) 0.807 ± 0.003(5) 0.499 ± 0.000(5) 0.499 ± 0.000(1) 1.0 0.194 ± 0.002(5) 0.810 ± 0.003(5) 0.496 ± 0.000(5) 0.438 ± 0.000(1) Mean 0.506 ± 0.003(5) 0.740 ± 0.015(5) 0.582 ± 0.000(5) 0.600 ± 0.000(1)

Clean ratios 5.0%

Noise Level Forward GLC L2R MLC 0.0 0.810 ± 0.003(5) 0.810 ± 0.002(5) 0.761 ± 0.000(5) 0.761 ± 0.000(1) 0.1 0.801 ± 0.008(5) 0.813 ± 0.004(5) 0.742 ± 0.000(5) 0.770 ± 0.000(1) 0.2 0.789 ± 0.007(5) 0.806 ± 0.004(5) 0.607 ± 0.000(5) 0.763 ± 0.000(1) 0.3 0.761 ± 0.007(5) 0.788 ± 0.009(5) 0.500 ± 0.000(5) 0.733 ± 0.000(1) 0.4 0.701 ± 0.011(5) 0.661 ± 0.124(5) 0.499 ± 0.000(5) 0.718 ± 0.000(1) 0.5 0.559 ± 0.009(5) 0.572 ± 0.071(5) 0.499 ± 0.000(5) 0.698 ± 0.000(1) 0.6 0.354 ± 0.009(5) 0.653 ± 0.101(5) 0.499 ± 0.000(5) 0.684 ± 0.000(1) 0.7 0.259 ± 0.012(5) 0.788 ± 0.011(5) 0.499 ± 0.000(5) 0.671 ± 0.000(1) 0.8 0.224 ± 0.010(5) 0.805 ± 0.005(5) 0.499 ± 0.000(5) 0.635 ± 0.000(1) 0.9 0.206 ± 0.009(5) 0.812 ± 0.008(5) 0.499 ± 0.000(5) 0.616 ± 0.000(1) 1.0 0.192 ± 0.004(5) 0.811 ± 0.002(5) 0.499 ± 0.000(5) 0.572 ± 0.000(1) Mean 0.514 ± 0.005(5) 0.756 ± 0.021(5) 0.555 ± 0.000(5) 0.693 ± 0.000(1) 5) 0.464 ± 0.184(5) 0.696 ± 0.007(5) 0.732 ± 0.000(1) 2 0.377 ± 0.009(5) 0.343 ± 0.124(5) 0.616 ± 0.014(5) 0.752 ± 0.000(1) 3 0.230 ± 0.015(5) 0.195 ± 0.052(5) 0.466 ± 0.042(5) 0.693 ± 0.000(1) Mean 0.383 ± 0.008(5) 0.334 ± 0.119(5) 0.593 ± 0.017(5) 0.726 ± 0.000(1)

Clean ratios 1.0%

Weak Classifier Forward GLC L2R MLC 1 0.542 ± 0.013(5) 0.589 ± 0.013(5) 0.721 ± 0.010(5) 0.879 ± 0.000(1) 2 0.404 ± 0.025(5) 0.446 ± 0.015(5) 0.662 ± 0.013(5) 0.881 ± 0.000(1) 3 0.254 ± 0.025(5) 0.317 ± 0.019(5) 0.470 ± 0.015(5) 0.885 ± 0.000(1) Mean 0.400 ± 0.018(5) 0.451 ± 0.011(5) 0.618 ± 0.011(5) 0.882 ± 0.000(1)

Clean ratios 5.0%

Weak Classifier Forward GLC L2R MLC 1 0.555 ± 0.005(5) 0.612 ± 0.005(5) 0.720 ± 0.016(5) 0.922 ± 0.000(1) 2 0.451 ± 0.044(5) 0.567 ± 0.014(5) 0.635 ± 0.015(5) 0.924 ± 0.000(1) 3 0.304 ± 0.041(5) 0.524 ± 0.032(5) 0.480 ± 0.020(5) 0.918 ± 0.000(1) Mean 0.437 ± 0.022(5) 0.568 ± 0.014(5) 0.612 ± 0.005(5) 0.921 ± 0.000(1)

:

http://www.di.unipi.it/ ˜gulli/AG_corpus_of_news_articles.html

https://pytorch.org

Figures and Tables:

Figure 1: MLC Architecture. Nodes in gray denotes the noisy training examples and green ones denotes the clean ones.The meta model g α (in orange) takes in a noisy example label pair and tries to generate a "corrected" label which will be treated as the correct labels to train the main model f w (in blue). Thus the trained main model depends on the corrected labels y c , hence further depends on the parameters of the meta model. The trained main model then will be tested on a separate true clean set, whose loss is to be minimized. Note that when minimizing the loss on the clean examples, the parameters for the main models are not changed, only to let the loss signal on the clean examples to propogate back to the meta network, thus making g α generate better corrections for x, y . In practice, we won't be able to always get the trained main model to evaluate on the clean examples, thus k-step SGD ahead updated version of the main model is used as a "trained" model.

Figure 2: Results on Twitter. All numbers reported are accuracies on the test set.

Figure 3: Results on MNIST. All numbers reported are accuracies on the test set. Best results in terms of mean accuracies are printed in black.

Figure7: Results on IMDB. All numbers reported are accuracies on the test set. For references, using gold data only to train a model yields test accurucies of (0.541?, 0.647? and 0.741?, for three gold data ratios respectively.

Overview of network architecture used for main model and meta model

Table content: <table><tr><td>Data set</td><td>Meta network</td><td>Main network</td></tr><tr><td>Twitter, SST</td><td colspan="2">Embedding Avg Embedding Avg</td></tr><tr><td>MNIST</td><td>3-layer MLP</td><td>3-layer MLP</td></tr><tr><td>IMDB</td><td>1-layer LSTM</td><td>1-layer LSTM</td></tr><tr><td>AG, Amazon2, Amazon5, Yelp2, Yelp5, DBpedia</td><td>BERT-base</td><td>BERT-base</td></tr></table>

Mean accuracies over an extensive set of experimental configurations.

Table content: <table><tr><td>Each cell represents</td></tr></table>

Error rates comparison on 6 large text data sets

Table content: <table><tr><td>Datasets</td><td colspan="6">AG Yelp-2 Yelp-5 Amazon-2 Amazon-5 DBpedia</td></tr><tr><td colspan="4">Fully supervised (# labeled examples) (120k) (560k) (650k)</td><td>(3.6m)</td><td>(3m)</td><td>(560k)</td></tr><tr><td>BERTLARGE (Xie et al., 2019)</td><td>-</td><td>1.89</td><td>29.32</td><td>2.63</td><td>34.17</td><td>0.64</td></tr><tr><td>SSL (# labeled examples)</td><td/><td>(20)</td><td>(2.5k)</td><td>(20)</td><td>(2.5k)</td><td>(140)</td></tr><tr><td>BERTBASE-512</td><td>-</td><td>13.60</td><td>41.00</td><td>26.75</td><td>44.09</td><td>2.58</td></tr><tr><td>BERTLARGE-512</td><td>-</td><td>10.55</td><td>38.90</td><td>15.54</td><td>42.30</td><td>1.68</td></tr><tr><td>WSL (# labeled examples, p = 0.6)</td><td>(60)</td><td>(20)</td><td>(2.5k)</td><td>(20)</td><td>(2.5k)</td><td>(140)</td></tr><tr><td>GLC -BERTBASE-128</td><td>18.74</td><td>8.87</td><td>44.79</td><td>10.19</td><td>48.08</td><td>3.10</td></tr><tr><td>L2R -BERTBASE-128</td><td>8.37</td><td>10.00</td><td>38.74</td><td>10.77</td><td>42.93</td><td>2.08</td></tr><tr><td>MLC -BERTBASE-128</td><td>9.25</td><td>8.18</td><td>37.69</td><td>9.54</td><td>42.53</td><td>1.70</td></tr></table>

IMDB. Figure7presents the detailed results on all three noisy settings...

Table content: <table><tr><td/><td>1.0</td><td>SST, unif, 0.1% clean</td><td/><td>1.0</td><td>SST, unif, 1.0% clean</td><td/><td>1.0</td><td>SST, unif, 5.0% clean</td></tr><tr><td>Test accuracy Test accuracy Test accuracy Test accuracy Test accuracy</td><td>0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0</td><td>0.00 0.25 0.50 0.75 1.00 Noise level Forward GLC L2R MLC IMDB, unif, 0.1% clean 0.00 0.25 0.50 0.75 1.00 Noise level SST, flip, 0.1% clean Forward GLC L2R MLC 0.00 0.25 0.50 0.75 1.00 Noise level Forward GLC L2R 1 2 3 Weak classifier index SST, weak, 0.1% clean Forward GLC L2R MLC 0.00 0.25 0.50 0.75 1.00 Noise level Forward GLC L2R IMDB, flip, 0.1% clean</td><td>Test accuracy Test accuracy Test accuracy Test accuracy Test accuracy</td><td>0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0</td><td>0.00 0.25 0.50 0.75 1.00 Noise level Forward GLC L2R MLC IMDB, unif, 1.0% clean 0.00 0.25 0.50 0.75 1.00 Noise level SST, flip, 1.0% clean Forward GLC L2R MLC 0.00 0.25 0.50 0.75 1.00 Noise level Forward GLC L2R 1 2 3 Weak classifier index SST, weak, 1.0% clean Forward GLC L2R MLC 0.00 0.25 0.50 0.75 1.00 Noise level Forward GLC L2R IMDB, flip, 1.0% clean</td><td>Test accuracy Test accuracy Test accuracy Test accuracy Test accuracy</td><td>0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 0.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0</td><td>0.00 0.25 0.50 0.75 1.00 Noise level Forward IMDB, unif, 5.0% clean GLC L2R MLC 0.00 0.25 0.50 0.75 1.00 Noise level SST, flip, 5.0% clean Forward GLC L2R 0.00 0.25 0.50 0.75 1.00 Noise level Forward L2R MLC 1 2 Weak classifier index 3 Forward GLC L2R MLC SST, weak, 5.0% clean Noise level 0.00 0.25 0.50 0.75 1.00 GLC IMDB, flip, 5.0% clean</td></tr><tr><td colspan="9">Figure 6: Results on SST. All numbers reported are accuracies on the test set. For references, using</td></tr><tr><td colspan="9">gold data only to train a model yields test accurucies of 0.541?, 0.647? and 0.741?, for three gold</td></tr><tr><td colspan="3">data ratios respectively.</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="3">B COMPLETE TABLES</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="3">B.1 TWITTER</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="3">B.2 SST</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="3">B.3 IMDB</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="3">B.4 MNIST</td><td/><td/><td/><td/><td/><td/></tr></table>

Detailed test set accuracies on TWITTER with different ratios of clean data and label noise levels from UNIF. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td colspan="2">1.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.490 ± 0.144(5)</td><td>0.835 ± 0.042(5)</td><td>0.859 ± 0.000(5)</td><td>0.827 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.682 ± 0.048(5)</td><td>0.835 ± 0.034(5)</td><td>0.860 ± 0.000(5)</td><td>0.766 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.693 ± 0.025(5)</td><td>0.813 ± 0.039(5)</td><td>0.854 ± 0.000(5)</td><td>0.742 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.692 ± 0.023(5)</td><td>0.800 ± 0.036(5)</td><td>0.846 ± 0.000(5)</td><td>0.729 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.608 ± 0.042(5)</td><td>0.793 ± 0.017(5)</td><td>0.841 ± 0.000(5)</td><td>0.721 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.575 ± 0.038(5)</td><td>0.749 ± 0.031(5)</td><td>0.822 ± 0.000(5)</td><td>0.705 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.481 ± 0.047(5)</td><td>0.707 ± 0.024(5)</td><td>0.815 ± 0.000(5)</td><td>0.705 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.414 ± 0.040(5)</td><td>0.613 ± 0.040(5)</td><td>0.766 ± 0.000(5)</td><td>0.678 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.333 ± 0.049(5)</td><td>0.468 ± 0.036(5)</td><td>0.745 ± 0.000(5)</td><td>0.682 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.156 ± 0.036(5)</td><td>0.325 ± 0.034(5)</td><td>0.649 ± 0.000(5)</td><td>0.679 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.044 ± 0.011(5)</td><td>0.204 ± 0.012(5)</td><td>0.311 ± 0.000(5)</td><td>0.695 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.470 ± 0.014(5)</td><td>0.649 ± 0.022(5)</td><td>0.761 ± 0.000(5)</td><td>0.721 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">5.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.538 ± 0.098(5)</td><td>0.863 ± 0.023(5)</td><td>0.867 ± 0.000(5)</td><td>0.855 ± 0.000(1)</td></tr><tr><td>0.1</td><td colspan="2">0.665 ± 0.096(5) 0.867 ± 0.003(5)</td><td>0.862 ± 0.000(5)</td><td>0.831 ± 0.000(1)</td></tr><tr><td>0.2</td><td colspan="2">0.714 ± 0.045(5) 0.855 ± 0.004(5)</td><td>0.843 ± 0.000(5)</td><td>0.815 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.687 ± 0.019(5)</td><td>0.838 ± 0.009(5)</td><td>0.847 ± 0.000(5)</td><td>0.795 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.625 ± 0.019(5)</td><td>0.832 ± 0.006(5)</td><td>0.841 ± 0.000(5)</td><td>0.793 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.614 ± 0.043(5)</td><td>0.817 ± 0.006(5)</td><td>0.830 ± 0.000(5)</td><td>0.777 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.541 ± 0.013(5)</td><td>0.796 ± 0.005(5)</td><td>0.807 ± 0.000(5)</td><td>0.778 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.460 ± 0.027(5)</td><td>0.755 ± 0.006(5)</td><td>0.779 ± 0.000(5)</td><td>0.770 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.359 ± 0.044(5)</td><td>0.698 ± 0.019(5)</td><td>0.746 ± 0.000(5)</td><td>0.769 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.189 ± 0.040(5)</td><td>0.622 ± 0.012(5)</td><td>0.673 ± 0.000(5)</td><td>0.753 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.092 ± 0.020(5)</td><td>0.557 ± 0.017(5)</td><td>0.305 ± 0.000(5)</td><td>0.759 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.498 ± 0.015(5)</td><td>0.773 ± 0.003(5)</td><td>0.764 ± 0.000(5)</td><td>0.791 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">10.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>-</td><td>0.866 ± 0.012(5)</td><td>0.869 ± 0.004(5)</td><td>0.867 ± 0.000(1)</td></tr><tr><td>0.1</td><td>-</td><td>0.855 ± 0.013(5)</td><td>0.862 ± 0.002(5)</td><td>0.859 ± 0.000(1)</td></tr><tr><td>0.2</td><td>-</td><td>0.854 ± 0.003(5)</td><td>0.858 ± 0.003(5)</td><td>0.846 ± 0.000(1)</td></tr><tr><td>0.3</td><td>-</td><td>0.842 ± 0.007(5)</td><td>0.848 ± 0.005(5)</td><td>0.837 ± 0.000(1)</td></tr><tr><td>0.4</td><td>-</td><td>0.832 ± 0.011(5)</td><td>0.842 ± 0.004(5)</td><td>0.831 ± 0.000(1)</td></tr><tr><td>0.5</td><td>-</td><td>0.825 ± 0.005(5)</td><td>0.829 ± 0.003(5)</td><td>0.827 ± 0.000(1)</td></tr><tr><td>0.6</td><td>-</td><td>0.809 ± 0.005(5)</td><td>0.809 ± 0.003(5)</td><td>0.817 ± 0.000(1)</td></tr><tr><td>0.7</td><td>-</td><td>0.790 ± 0.008(5)</td><td>0.784 ± 0.006(5)</td><td>0.818 ± 0.000(1)</td></tr><tr><td>0.8</td><td>-</td><td>0.766 ± 0.012(5)</td><td>0.737 ± 0.008(5)</td><td>0.810 ± 0.000(1)</td></tr><tr><td>0.9</td><td>-</td><td>0.732 ± 0.014(5)</td><td>0.666 ± 0.006(5)</td><td>0.805 ± 0.000(1)</td></tr><tr><td>1.0</td><td>-</td><td>0.713 ± 0.012(5)</td><td>0.302 ± 0.018(5)</td><td>0.796 ± 0.000(1)</td></tr><tr><td>Mean</td><td>-</td><td>0.808 ± 0.004(5)</td><td>0.764 ± 0.002(5)</td><td>0.828 ± 0.000(1)</td></tr></table>

Detailed test set accuracies on TWITTER with different ratios of clean data and label noise levels from FLIP. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td colspan="2">1.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.490 ± 0.144(5)</td><td>0.835 ± 0.042(5)</td><td>0.863 ± 0.000(5)</td><td>0.811 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.529 ± 0.073(5)</td><td>0.837 ± 0.032(5)</td><td>0.855 ± 0.000(5)</td><td>0.783 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.549 ± 0.086(5)</td><td>0.828 ± 0.046(5)</td><td>0.854 ± 0.000(5)</td><td>0.763 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.542 ± 0.076(5)</td><td>0.816 ± 0.048(5)</td><td>0.841 ± 0.000(5)</td><td>0.742 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.553 ± 0.039(5)</td><td>0.811 ± 0.035(5)</td><td>0.828 ± 0.000(5)</td><td>0.739 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.453 ± 0.037(5)</td><td>0.816 ± 0.025(5)</td><td>0.817 ± 0.000(5)</td><td>0.728 ± 0.000(1)</td></tr><tr><td>0.6</td><td colspan="2">0.386 ± 0.080(5) 0.814 ± 0.013(5)</td><td>0.799 ± 0.000(5)</td><td>0.712 ± 0.000(1)</td></tr><tr><td>0.7</td><td colspan="2">0.292 ± 0.078(5) 0.782 ± 0.061(5)</td><td>0.777 ± 0.000(5)</td><td>0.698 ± 0.000(1)</td></tr><tr><td>0.8</td><td colspan="2">0.320 ± 0.094(5) 0.776 ± 0.040(5)</td><td>0.704 ± 0.000(5)</td><td>0.712 ± 0.000(1)</td></tr><tr><td>0.9</td><td colspan="2">0.264 ± 0.100(5) 0.747 ± 0.082(5)</td><td>0.596 ± 0.000(5)</td><td>0.708 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.103 ± 0.098(5)</td><td>0.654 ± 0.129(5)</td><td>0.217 ± 0.000(5)</td><td>0.669 ± 0.000(1)</td></tr><tr><td>Mean</td><td colspan="2">0.407 ± 0.035(5) 0.792 ± 0.042(5)</td><td>0.741 ± 0.000(5)</td><td>0.733 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">5.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.538 ± 0.098(5)</td><td>0.863 ± 0.023(</td><td/><td/></tr></table>

Detailed test set accuracies on TWITTER with different ratios of clean data and label noise levels from WEAK. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td>1.0%</td><td/><td/></tr><tr><td>Weak Classifier</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>1</td><td>0.260 ± 0.045(</td><td/><td/><td/></tr></table>

Detailed test set accuracies on SST with different ratios of clean data and label noise levels from UNIF. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td>0.1%</td><td/><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr></table>

Detailed test set accuracies on SST with different ratios of clean data and label noise levels from FLIP. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td colspan="2">0.1%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.810 ± 0.002(5)</td><td colspan="2">0.812 ± 0.001(5) 0.747 ± 0.000(5)</td><td>0.669 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.802 ± 0.006(5)</td><td colspan="2">0.812 ± 0.005(5) 0.690 ± 0.004(5)</td><td>0.646 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.789 ± 0.003(5)</td><td colspan="2">0.803 ± 0.005(5) 0.640 ± 0.000(5)</td><td>0.617 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.757 ± 0.003(5)</td><td colspan="2">0.777 ± 0.007(5) 0.535 ± 0.000(5)</td><td>0.601 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.685 ± 0.007(5)</td><td>0.675 ± 0.087(5)</td><td>0.510 ± 0.000(5)</td><td>0.505 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.505 ± 0.009(5)</td><td>0.501 ± 0.003(5)</td><td>0.504 ± 0.000(5)</td><td>0.490 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.316 ± 0.009(5)</td><td colspan="2">0.628 ± 0.098(5) 0.498 ± 0.001(5)</td><td>0.484 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.249 ± 0.009(5)</td><td colspan="2">0.776 ± 0.012(5) 0.482 ± 0.000(5)</td><td>0.475 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.221 ± 0.008(5)</td><td colspan="2">0.797 ± 0.007(5) 0.469 ± 0.000(5)</td><td>0.463 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.204 ± 0.012(5)</td><td colspan="2">0.807 ± 0.006(5) 0.479 ± 0.000(5)</td><td>0.334 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.190 ± 0.002(5)</td><td colspan="2">0.812 ± 0.001(5) 0.499 ± 0.000(5)</td><td>0.301 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.503 ± 0.002(5)</td><td colspan="2">0.746 ± 0.016(5) 0.550 ± 0.000(5)</td><td>0.508 ± 0.000(1)</td></tr></table>

Detailed test set accuracies on SST with different ratios of clean data and label noise levels from WEAK. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td colspan="2">0.1%</td><td/></tr><tr><td>Weak Classifier</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>1</td><td>0.704 ± 0.002(5)</td><td colspan="2">0.708 ± 0.002(5) 0.673 ± 0.014(5)</td><td>0.670 ± 0.000(1)</td></tr><tr><td>2</td><td>0.607 ± 0.001(5)</td><td colspan="2">0.638 ± 0.013(5) 0.569 ± 0.019(5)</td><td>0.541 ± 0.000(1)</td></tr><tr><td>3</td><td>0.564 ± 0.010(5)</td><td>0.522 ± 0.026(5)</td><td>0.512 ± 0.010(5)</td><td>0.523 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.625 ± 0.003(5)</td><td>0.623 ± 0.007(5)</td><td>0.585 ± 0.010(5)</td><td>0.578 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">1.0%</td><td/></tr><tr><td>Weak Classifier</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>1</td><td>0.704 ± 0.003(5)</td><td colspan="2">0.711 ± 0.002(5) 0.681 ± 0.003(5)</td><td>0.630 ± 0.000(1)</td></tr><tr><td>2</td><td>0.610 ± 0.001(5)</td><td colspan="2">0.637 ± 0.004(5) 0.588 ± 0.010(5)</td><td>0.625 ± 0.000(1)</td></tr><tr><td>3</td><td>0.568 ± 0.008(5)</td><td>0.536 ± 0.022(5)</td><td colspan="2">0.523 ± 0.012(5) 0.620 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.627 ± 0.003(5)</td><td colspan="2">0.628 ± 0.009(5) 0.597 ± 0.008(5)</td><td>0.625 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">5.0%</td><td/></tr><tr><td>Weak Classifier</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>1</td><td>0.708 ± 0.003(5)</td><td>0.708 ± 0.006(5)</td><td colspan="2">0.682 ± 0.001(5) 0.730 ± 0.000(1)</td></tr><tr><td>2</td><td>0.618 ± 0.003(5)</td><td>0.558 ± 0.003(5)</td><td colspan="2">0.588 ± 0.005(5) 0.688 ± 0.000(1)</td></tr><tr><td>3</td><td>0.594 ± 0.021(5)</td><td>0.521 ± 0.012(5)</td><td colspan="2">0.515 ± 0.014(5) 0.690 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.640 ± 0.008(5)</td><td>0.595 ± 0.005(5)</td><td colspan="2">0.595 ± 0.005(5) 0.703 ± 0.000(1)</td></tr></table>

Detailed test set accuracies on IMDB with different ratios of clean data and label noise levels from UNIF. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td>0.1%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td></tr><tr><td>0.0</td><td>0.860 ± 0.008(5)</td><td>0.859 ± 0.003(5)</td><td>0.861 ± 0.000(5)</td></tr><tr><td>0.1</td><td>0.856 ± 0.006(5)</td><td>0.842 ± 0.009(5)</td><td>0.816 ± 0.015(5)</td></tr><tr><td>0.2</td><td>0.849 ± 0.003(5)</td><td>0.820 ± 0.020(5)</td><td>0.782 ± 0.000(5)</td></tr><tr><td>0.3</td><td>0.835 ± 0.008(5)</td><td>0.680 ± 0.120(5)</td><td>0.770 ± 0.000(5)</td></tr><tr><td>0.4</td><td>0.815 ± 0.015(5)</td><td>0.786 ± 0.038(5)</td><td>0.775 ± 0.000(5)</td></tr><tr><td>0.5</td><td>0.791 ± 0.009(5)</td><td>0.735 ± 0.116(5)</td><td>0.697 ± 0.000(5)</td></tr><tr><td>0.6</td><td>0.748 ± 0.010(5)</td><td>0.711 ± 0.058(5)</td><td>0.657 ± 0.035(5)</td></tr><tr><td>0.7</td><td>0.693 ± 0.018(5)</td><td>0.645 ± 0.144(5)</td><td>0.559 ± 0.000(5)</td></tr><tr><td>0.8</td><td>0.607 ± 0.017(5)</td><td>0.622 ± 0.050(5)</td><td>0.594 ± 0.000(5)</td></tr><tr><td>0.9</td><td>0.540 ± 0.007(5)</td><td>0.527 ± 0.039(5)</td><td>0.541 ± 0.000(5)</td></tr><tr><td>1.0</td><td>0.496 ± 0.006(5)</td><td>0.510 ± 0.011(5)</td><td>0.504 ± 0.000(5)</td></tr><tr><td>Mean</td><td>0.735 ± 0.003(5)</td><td>0.703 ± 0.031(5)</td><td>0.687 ± 0.004(5)</td></tr><tr><td>Clean ratios</td><td/><td>1.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td></tr><tr><td>0.0</td><td>0.865 ± 0.004(5)</td><td>0.848 ± 0.003(5)</td><td>0.857 ± 0.000(5)</td></tr><tr><td>0.1</td><td>0.858 ± 0.003(5)</td><td>0.828 ± 0.025(5)</td><td>0.842 ± 0.000(5)</td></tr><tr><td>0.2</td><td>0.821 ± 0.044(5)</td><td>0.758 ± 0.119(5)</td><td>0.824 ± 0.000(5)</td></tr><tr><td>0.3</td><td>0.828 ± 0.010(5)</td><td>0.833 ± 0.004(5)</td><td>0.810 ± 0.000(5)</td></tr><tr><td>0.4</td><td>0.823 ± 0.007(5)</td><td>0.788 ± 0.033(5)</td><td>0.771 ± 0.000(5)</td></tr><tr><td>0.5</td><td>0.782 ± 0.013(5)</td><td>0.787 ± 0.020(5)</td><td>0.744 ± 0.000(5)</td></tr><tr><td>0.6</td><td>0.709 ± 0.050(5)</td><td>0.758 ± 0.004(5)</td><td>0.676 ± 0.000(5)</td></tr><tr><td>0.7</td><td>0.697 ± 0.019(5)</td><td>0.728 ± 0.005(5)</td><td>0.632 ± 0.000(5)</td></tr><tr><td>0.8</td><td>0.597 ± 0.008(5)</td><td>0.644 ± 0.037(5)</td><td>0.559 ± 0.000(5)</td></tr><tr><td>0.9</td><td>0.539 ± 0.005(5)</td><td>0.584 ± 0.013(5)</td><td>0.551 ± 0.000(5)</td></tr><tr><td>1.0</td><td>0.499 ± 0.007(5)</td><td>0.563 ± 0.017(5)</td><td>0.507 ± 0.000(5)</td></tr><tr><td>Mean</td><td>0.729 ± 0.005(5)</td><td>0.738 ± 0.013(5)</td><td>0.707 ± 0.000(5)</td></tr><tr><td>Clean ratios</td><td/><td>5.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td></tr><tr><td>0.0</td><td>0.859 ± 0.005(5)</td><td>0.855 ± 0.009(5)</td><td>0.860 ± 0.000(5)</td></tr><tr><td>0.1</td><td>0.849 ± 0.009(5)</td><td>0.793 ± 0.076(5)</td><td>0.835 ± 0.000(5)</td></tr><tr><td>0.2</td><td>0.848 ± 0.003(5)</td><td>0.828 ± 0.017(5)</td><td>0.821 ± 0.000(5)</td></tr><tr><td>0.3</td><td>0.826 ± 0.015(5)</td><td>0.832 ± 0.013(5)</td><td>0.813 ± 0.000(5)</td></tr><tr><td>0.4</td><td>0.814 ± 0.010(5)</td><td>0.797 ± 0.025(5)</td><td>0.786 ± 0.000(5)</td></tr><tr><td>0.5</td><td>0.789 ± 0.012(5)</td><td>0.804 ± 0.009(5)</td><td>0.731 ± 0.000(5)</td></tr><tr><td>0.6</td><td>0.739 ± 0.032(5)</td><td>0.762 ± 0.052(5)</td><td>0.700 ± 0.000(5)</td></tr><tr><td>0.7</td><td>0.726 ± 0.009(5)</td><td>0.767 ± 0.009(5)</td><td>0.637 ± 0.000(5)</td></tr><tr><td>0.8</td><td>0.631 ± 0.016(5)</td><td>0.741 ± 0.011(5)</td><td>0.585 ± 0.000(5)</td></tr><tr><td>0.9</td><td>0.566 ± 0.009(5)</td><td>0.692 ± 0.036(5)</td><td>0.551 ± 0.000(5)</td></tr><tr><td>1.0</td><td>0.516 ± 0.008(5)</td><td>0.677 ± 0.041(5)</td><td>0.497 ± 0.000(5)</td></tr><tr><td>Mean</td><td>0.742 ± 0.004(5)</td><td>0.777 ± 0.016(5)</td><td>0.711 ± 0.000(5)</td></tr></table>

Detailed test set accuracies on IMDB with different ratios of clean data and label noise levels from FLIP. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td>0.1%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td></tr><tr><td>0.0</td><td>0.860 ± 0.008(5)</td><td>0.859 ± 0.003(5)</td><td>0.861 ± 0.000(5)</td></tr><tr><td>0.1</td><td>0.849 ± 0.003(5)</td><td>0.820 ± 0.020(5)</td><td>0.811 ± 0.030(5)</td></tr><tr><td>0.2</td><td>0.815 ± 0.015(5)</td><td>0.786 ± 0.038(5)</td><td>0.775 ± 0.000(5)</td></tr><tr><td>0.3</td><td>0.748 ± 0.010(5)</td><td>0.711 ± 0.058(5)</td><td>0.675 ± 0.000(5)</td></tr><tr><td>0.4</td><td>0.607 ± 0.017(5)</td><td>0.622 ± 0.050(5)</td><td>0.594 ± 0.000(5)</td></tr><tr><td>0.5</td><td>0.496 ± 0.006(5)</td><td>0.510 ± 0.011(5)</td><td>0.504 ± 0.000(5)</td></tr><tr><td>0.6</td><td>0.392 ± 0.017(5)</td><td>0.635 ± 0.021(5)</td><td>0.463 ± 0.021(5)</td></tr><tr><td>0.7</td><td>0.260 ± 0.012(5)</td><td>0.701 ± 0.060(5)</td><td>0.448 ± 0.000(5)</td></tr><tr><td>0.8</td><td>0.186 ± 0.012(5)</td><td>0.806 ± 0.014(5)</td><td>0.399 ± 0.000(5)</td></tr><tr><td>0.9</td><td>0.154 ± 0.005(5)</td><td>0.825 ± 0.032(5)</td><td>0.408 ± 0.000(5)</td></tr><tr><td>1.0</td><td>0.136 ± 0.004(5)</td><td>0.851 ± 0.012(5)</td><td>0.421 ± 0.000(5)</td></tr><tr><td>Mean</td><td>0.500 ± 0.003(5)</td><td>0.739 ± 0.011(5)</td><td>0.578 ± 0.004(5)</td></tr><tr><td>Clean ratios</td><td/><td>1.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td></tr><tr><td>0.0</td><td>0.865 ± 0.004(5)</td><td>0.848 ± 0.003(5)</td><td>0.857 ± 0.000(5)</td></tr><tr><td>0.1</td><td>0.821 ± 0.044(5)</td><td>0.758 ± 0.119(5)</td><td>0.823 ± 0.000(5)</td></tr><tr><td>0.2</td><td>0.823 ± 0.007(5)</td><td>0.788 ± 0.033(5)</td><td>0.771 ± 0.000(5)</td></tr><tr><td>0.3</td><td>0.709 ± 0.050(5)</td><td>0.758 ± 0.004(5)</td><td>0.682 ± 0.000(5)</td></tr><tr><td>0.4</td><td>0.597 ± 0.008(5)</td><td>0.644 ± 0.037(5)</td><td>0.559 ± 0.000(5)</td></tr><tr><td>0.5</td><td>0.499 ± 0.007(5)</td><td>0.563 ± 0.017(5)</td><td>0.507 ± 0.000(5)</td></tr><tr><td>0.6</td><td>0.403 ± 0.002(5)</td><td>0.666 ± 0.014(5)</td><td>0.493 ± 0.000(5)</td></tr><tr><td>0.7</td><td>0.278 ± 0.047(5)</td><td>0.742 ± 0.053(5)</td><td>0.500 ± 0.000(5)</td></tr><tr><td>0.8</td><td>0.186 ± 0.010(5)</td><td>0.806 ± 0.019(5)</td><td>0.500 ± 0.000(5)</td></tr><tr><td>0.9</td><td>0.169 ± 0.027(5)</td><td>0.805 ± 0.036(5)</td><td>0.500 ± 0.000(5)</td></tr><tr><td>1.0</td><td>0.142 ± 0.003(5)</td><td>0.767 ± 0.125(5)</td><td>0.500 ± 0.000(5)</td></tr><tr><td>Mean</td><td>0.499 ± 0.007(5)</td><td>0.741 ± 0.018(5)</td><td>0.608 ± 0.000(5)</td></tr><tr><td>Clean ratios</td><td/><td>5.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td></tr><tr><td>0.0</td><td>0.859 ± 0.005(5)</td><td>0.855 ± 0.009(5)</td><td>0.860 ± 0.000(5)</td></tr><tr><td>0.1</td><td>0.848 ± 0.003(5)</td><td>0.828 ± 0.017(5)</td><td>0.828 ± 0.000(5)</td></tr><tr><td>0.2</td><td>0.814 ± 0.010(5)</td><td>0.797 ± 0.025(5)</td><td>0.786 ± 0.000(5)</td></tr><tr><td>0.3</td><td>0.739 ± 0.032(5)</td><td>0.762 ± 0.052(5)</td><td>0.699 ± 0.000(5)</td></tr><tr><td>0.4</td><td>0.631 ± 0.016(5)</td><td>0.741 ± 0.011(5)</td><td>0.585 ± 0.000(5)</td></tr><tr><td>0.5</td><td>0.516 ± 0.008(5)</td><td>0.677 ± 0.041(5)</td><td>0.497 ± 0.000(5)</td></tr><tr><td>0.6</td><td>0.435 ± 0.009(5)</td><td>0.735 ± 0.013(5)</td><td>0.500 ± 0.000(5)</td></tr><tr><td>0.7</td><td>0.284 ± 0.018(5)</td><td>0.773 ± 0.024(5)</td><td>0.500 ± 0.000(5)</td></tr><tr><td>0.8</td><td>0.209 ± 0.006(5)</td><td>0.830 ± 0.004(5)</td><td>0.500 ± 0.000(5)</td></tr><tr><td>0.9</td><td>0.164 ± 0.008(5)</td><td>0.835 ± 0.009(5)</td><td>0.500 ± 0.000(5)</td></tr><tr><td>1.0</td><td>0.143 ± 0.004(5)</td><td>0.849 ± 0.013(5)</td><td>0.481 ± 0.000(5)</td></tr><tr><td>Mean</td><td>0.513 ± 0.007(5)</td><td>0.789 ± 0.008(5)</td><td>0.612 ± 0.000(5)</td></tr></table>

Detailed test set accuracies on IMDB with different ratios of clean data and label noise levels from WEAK. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td>0.1%</td></tr><tr><td>Weak Classifier</td><td>MLC</td></tr><tr><td>1</td><td>0.682 ± 0.029(5)</td></tr><tr><td>2</td><td>0.623 ± 0.004(5)</td></tr><tr><td>3</td><td>0.546 ± 0.014(5)</td></tr><tr><td>Mean</td><td>0.617 ± 0.010(5)</td></tr><tr><td>Clean ratios</td><td>1.0%</td></tr><tr><td>Weak Classifier</td><td>MLC</td></tr><tr><td>1</td><td>0.703 ± 0.009(5)</td></tr><tr><td>2</td><td>0.611 ± 0.015(5)</td></tr><tr><td>3</td><td>0.536 ± 0.018(5)</td></tr><tr><td>Mean</td><td>0.616 ± 0.014(5)</td></tr><tr><td>Clean ratios</td><td>5.0%</td></tr><tr><td>Weak Classifier</td><td>MLC</td></tr><tr><td>1</td><td>0.700 ± 0.007(5)</td></tr><tr><td>2</td><td>0.635 ± 0.003(5)</td></tr><tr><td>3</td><td>± 0.010(5)</td></tr><tr><td>Mean</td><td>0.636 ± 0.004(5)</td></tr></table>

Detailed test set accuracies on MNIST with different ratios of clean data and label noise levels from UNIF. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td colspan="2">0.1%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.978 ± 0.002(5)</td><td>0.977 ± 0.002(3)</td><td>0.973 ± 0.005(5)</td><td>0.962 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.971 ± 0.002(5)</td><td>0.974 ± 0.001(3)</td><td>0.958 ± 0.008(5)</td><td>0.826 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.967 ± 0.003(5)</td><td>0.968 ± 0.002(3)</td><td>0.944 ± 0.031(5)</td><td>0.728 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.960 ± 0.002(5)</td><td>0.964 ± 0.002(3)</td><td>0.921 ± 0.030(5)</td><td>0.769 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.950 ± 0.001(5)</td><td>0.958 ± 0.002(3)</td><td>0.926 ± 0.016(5)</td><td>0.682 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.942 ± 0.003(5)</td><td>0.951 ± 0.005(3)</td><td>0.918 ± 0.015(5)</td><td>0.673 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.921 ± 0.008(5)</td><td>0.938 ± 0.007(3)</td><td>0.889 ± 0.033(5)</td><td>0.699 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.895 ± 0.010(5)</td><td>0.921 ± 0.001(3)</td><td>0.881 ± 0.013(5)</td><td>0.719 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.825 ± 0.020(5)</td><td>0.883 ± 0.005(3)</td><td>0.811 ± 0.028(5)</td><td>0.643 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.610 ± 0.073(5)</td><td>0.740 ± 0.033(3)</td><td>0.723 ± 0.023(5)</td><td>0.649 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.097 ± 0.022(5)</td><td>0.402 ± 0.054(3)</td><td>0.589 ± 0.029(5)</td><td>0.645 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.828 ± 0.007(5)</td><td>0.880 ± 0.003(3)</td><td>0.867 ± 0.007(5)</td><td>0.727 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">1.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.978 ± 0.001(5)</td><td>0.977 ± 0.002(3)</td><td>0.977 ± 0.002(5)</td><td>0.974 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.971 ± 0.001(5)</td><td>0.976 ± 0.002(3)</td><td>0.975 ± 0.001(5)</td><td>0.924 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.965 ± 0.002(5)</td><td>0.970 ± 0.002(3)</td><td>0.971 ± 0.001(5)</td><td>0.914 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.958 ± 0.004(5)</td><td>0.967 ± 0.002(3)</td><td>0.967 ± 0.001(5)</td><td>0.908 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.953 ± 0.002(5)</td><td>0.964 ± 0.004(3)</td><td>0.963 ± 0.002(5)</td><td>0.902 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.938 ± 0.004(5)</td><td>0.956 ± 0.005(3)</td><td>0.958 ± 0.002(5)</td><td>0.895 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.923 ± 0.006(5)</td><td>0.951 ± 0.004(3)</td><td>0.952 ± 0.001(5)</td><td>0.893 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.876 ± 0.008(5)</td><td>0.939 ± 0.001(3)</td><td>0.936 ± 0.003(5)</td><td>0.886 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.830 ± 0.015(5)</td><td>0.917 ± 0.008(3)</td><td>0.919 ± 0.002(5)</td><td>0.885 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.638 ± 0.030(5)</td><td>0.865 ± 0.007(3)</td><td>0.872 ± 0.012(5)</td><td>0.878 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.121 ± 0.009(5)</td><td>0.801 ± 0.022(3)</td><td>0.659 ± 0.014(5)</td><td>0.851 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.832 ± 0.003(5)</td><td>0.935 ± 0.001(3)</td><td>0.923 ± 0.002(5)</td><td>0.901 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">5.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.979 ± 0.002(5)</td><td>0.979 ± 0.001(3)</td><td>0.977 ± 0.002(5)</td><td>0.982 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.970 ± 0.002(5)</td><td>0.975 ± 0.001(3)</td><td>0.976 ± 0.001(5)</td><td>0.955 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.965 ± 0.002(5)</td><td>0.972 ± 0.002(3)</td><td>0.972 ± 0.002(5)</td><td>0.950 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.959 ± 0.006(5)</td><td>0.970 ± 0.003(3)</td><td>0.967 ± 0.001(5)</td><td>0.939 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.952 ± 0.003(5)</td><td>0.960 ± 0.001(3)</td><td>0.964 ± 0.002(5)</td><td>0.938 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.941 ± 0.003(5)</td><td>0.960 ± 0.002(3)</td><td>0.960 ± 0.002(5)</td><td>0.928 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.930 ± 0.006(5)</td><td>0.960 ± 0.003(3)</td><td>0.953 ± 0.003(5)</td><td>0.932 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.897 ± 0.011(5)</td><td>0.949 ± 0.001(3)</td><td>0.942 ± 0.003(5)</td><td>0.926 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.849 ± 0.020(5)</td><td>0.937 ± 0.007(3)</td><td>0.923 ± 0.002(5)</td><td>0.924 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.751 ± 0.051(5)</td><td>0.932 ± 0.005(3)</td><td>0.884 ± 0.003(5)</td><td>0.912 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.397 ± 0.024(5)</td><td>0.925 ± 0.002(3)</td><td>0.647 ± 0.029(5)</td><td>0.909 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.872 ± 0.009(5)</td><td>0.956 ± 0.001(3)</td><td>0.924 ± 0.003(5)</td><td>0.936 ± 0.000(1)</td></tr></table>

Detailed test set accuracies on MNIST with different ratios of clean data and label noise levels from FLIP. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td colspan="2">0.1%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.980 ± 0.000(2)</td><td>0.758 ± 0.381(4)</td><td>0.979 ± 0.002(5)</td><td>0.925 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.977 ± 0.003(2)</td><td>0.757 ± 0.380(4)</td><td>0.966 ± 0.008(5)</td><td>0.853 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.971 ± 0.006(2)</td><td>0.754 ± 0.379(4)</td><td>0.966 ± 0.003(5)</td><td>0.775 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.969 ± 0.009(2)</td><td>0.756 ± 0.380(4)</td><td>0.960 ± 0.006(5)</td><td>0.746 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.905 ± 0.017(2)</td><td>0.742 ± 0.372(4)</td><td>0.949 ± 0.010(5)</td><td>0.749 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.683 ± 0.076(2)</td><td>0.733 ± 0.368(4)</td><td>0.953 ± 0.006(5)</td><td>0.692 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.296 ± 0.060(2)</td><td>0.755 ± 0.379(4)</td><td>0.942 ± 0.004(5)</td><td>0.785 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.220 ± 0.101(2)</td><td>0.755 ± 0.379(4)</td><td>0.927 ± 0.016(5)</td><td>0.803 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.174 ± 0.106(2)</td><td>0.748 ± 0.376(4)</td><td>0.897 ± 0.024(5)</td><td>0.729 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.181 ± 0.119(2)</td><td>0.737 ± 0.369(4)</td><td>0.851 ± 0.032(5)</td><td>0.719 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.198 ± 0.006(2)</td><td>0.587 ± 0.283(4)</td><td>0.136 ± 0.067(5)</td><td>0.621 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.596 ± 0.028(2)</td><td>0.735 ± 0.368(4)</td><td>0.866 ± 0.007(5)</td><td>0.763 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">1.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.978 ± 0.001(2)</td><td>0.977 ± 0.001(4)</td><td>0.977 ± 0.002(5)</td><td>0.975 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.971 ± 0.005(2)</td><td>0.975 ± 0.001(4)</td><td>0.976 ± 0.001(5)</td><td>0.941 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.970 ± 0.008(2)</td><td>0.975 ± 0.002(4)</td><td>0.973 ± 0.002(5)</td><td>0.925 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.960 ± 0.015(2)</td><td>0.974 ± 0.003(4)</td><td>0.972 ± 0.001(5)</td><td>0.931 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.955 ± 0.023(2)</td><td>0.973 ± 0.004(4)</td><td>0.971 ± 0.002(5)</td><td>0.925 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.617 ± 0.007(2)</td><td>0.954 ± 0.033(4)</td><td>0.969 ± 0.003(5)</td><td>0.914 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.401 ± 0.069(2)</td><td>0.953 ± 0.038(4)</td><td>0.961 ± 0.003(5)</td><td>0.909 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.220 ± 0.111(2)</td><td>0.969 ± 0.006(4)</td><td>0.955 ± 0.005(5)</td><td>0.916 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.194 ± 0.109(2)</td><td>0.974 ± 0.001(4)</td><td>0.943 ± 0.003(5)</td><td>0.894 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.202 ± 0.098(2)</td><td>0.973 ± 0.001(4)</td><td>0.906 ± 0.007(5)</td><td>0.895 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.254 ± 0.041(2)</td><td>0.915 ± 0.044(4)</td><td>0.091 ± 0.014(5)</td><td>0.893 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.611 ± 0.035(2)</td><td>0.965 ± 0.006(4)</td><td>0.881 ± 0.001(5)</td><td>0.920 ± 0.000(1)</td></tr><tr><td>Clean ratios</td><td/><td colspan="2">5.0%</td><td/></tr><tr><td>Noise Level</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>0.0</td><td>0.980 ± 0.001(2)</td><td>0.978 ± 0.002(4)</td><td>0.977 ± 0.003(5)</td><td>0.979 ± 0.000(1)</td></tr><tr><td>0.1</td><td>0.974 ± 0.001(2)</td><td>0.976 ± 0.002(4)</td><td>0.976 ± 0.001(5)</td><td>0.963 ± 0.000(1)</td></tr><tr><td>0.2</td><td>0.974 ± 0.003(2)</td><td>0.975 ± 0.002(4)</td><td>0.974 ± 0.003(5)</td><td>0.956 ± 0.000(1)</td></tr><tr><td>0.3</td><td>0.968 ± 0.007(2)</td><td>0.973 ± 0.003(4)</td><td>0.973 ± 0.002(5)</td><td>0.945 ± 0.000(1)</td></tr><tr><td>0.4</td><td>0.927 ± 0.031(2)</td><td>0.975 ± 0.003(4)</td><td>0.972 ± 0.002(5)</td><td>0.943 ± 0.000(1)</td></tr><tr><td>0.5</td><td>0.758 ± 0.122(2)</td><td>0.971 ± 0.005(4)</td><td>0.969 ± 0.001(5)</td><td>0.942 ± 0.000(1)</td></tr><tr><td>0.6</td><td>0.262 ± 0.127(2)</td><td>0.969 ± 0.008(4)</td><td>0.966 ± 0.002(5)</td><td>0.936 ± 0.000(1)</td></tr><tr><td>0.7</td><td>0.221 ± 0.094(2)</td><td>0.974 ± 0.001(4)</td><td>0.961 ± 0.003(5)</td><td>0.929 ± 0.000(1)</td></tr><tr><td>0.8</td><td>0.210 ± 0.097(2)</td><td>0.972 ± 0.001(4)</td><td>0.944 ± 0.004(5)</td><td>0.927 ± 0.000(1)</td></tr><tr><td>0.9</td><td>0.206 ± 0.101(2)</td><td>0.972 ± 0.002(4)</td><td>0.909 ± 0.005(5)</td><td>0.923 ± 0.000(1)</td></tr><tr><td>1.0</td><td>0.194 ± 0.102(2)</td><td>0.969 ± 0.005(4)</td><td>0.125 ± 0.009(5)</td><td>0.922 ± 0.000(1)</td></tr><tr><td>Mean</td><td>0.607 ± 0.032(2)</td><td>0.973 ± 0.002(4)</td><td>0.886 ± 0.001(5)</td><td>0.942 ± 0.000(1)</td></tr></table>

Detailed test set accuracies on MNIST with different ratios of clean data and label noise levels from WEAK. (Best result in each row is printed in bold.)

Table content: <table><tr><td>Clean ratios</td><td/><td>0.1%</td><td/><td/></tr><tr><td>Weak Classifier</td><td>Forward</td><td>GLC</td><td>L2R</td><td>MLC</td></tr><tr><td>1</td><td>0.543 ± 0.007(</td><td/><td/><td/></tr></table>

Peer review (iclr2020):
This paper proposes a meta label correction approach for learning with noisy labels. They view the label correction procedure as a meta-process and then train a supervised model to fit the corrected labels generated by the meta-model. Some experiments and comparisons with SOTA are given, together with an ablation study. Pros: -This paper made an interesting contribution to use a meta-model for label correction and then train a supervised main model using the correction labels via a bi-level optimization problem. Cons: -The problem setting is a bit unclear to me. Weakly supervised learning and learning with noisy labels seem to be mixed up in this paper. Weakly supervised learning aims to learn from weak supervision, i.e., incomplete/ inexact/ inaccurate supervision (see more details in “A brief introduction to weakly supervised learning, National Science Review, 2018”), which is a more general concept. This paper only discussed learning from inaccurate supervision, thus stating learning with weak supervision may be a bit confusing. -In the introduction part, the authors reviewed two lines of prior works: re-weighting training instances and label correction. The literature review seems to be insufficient since there is another important research line that uses the sample selection bias trick. See “Jiang, L., Zhou, Z., Leung, T., Li, L., and Fei-Fei, L. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018” and “Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., and Sugiyama, M. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018”. -The strategy proposed in Sec 3.1 to convert a classifier to a label correction network seems a little suspicious to me. Can any further explanation or theoretical intuitions be given on how the hyperparameter lambda works? This would be very useful and make the paper more convincing. Overall the paper proposed an interesting idea to deal with noisy labels in a meta label correction way, but some unclear parts need to be clarified and improved. Further details: -Sec 2.2: there is a small mistake of the citation. -Sec 3.3: using a k-step look ahead SGD update as an estimate to the optimal model is good to me. But here k is set to 1500 and then decreased to 500, it seems to be too large and inefficient since the paper L2R also uses a similar trick for optimization and they only use k=1. How about the computation time? -In the experiments, MLC seems to be inferior in the UNIF case when using very small clean data and slightly outperforms baseline methods when using larger clean data. So how many clean data are used in tuning the baseline methods, since for L2R they only used 1000 clean data for all the experiments, and for Forward and GLC, they are also sensitive to the number of clean data used.