Venue: NAACL 2019

# An Effective Label Noise Model for DNN Text Classification

Ishan Jindal1, Daniel Pressel2, Brian Lester2, and Matthew Nokleby1

###### Abstract

Because large, human-annotated datasets suffer from labeling errors, it is crucial to be able to train deep neural networks in the presence of label noise. While training image classification models with label noise have received much attention, training text classification models have not. In this paper, we propose an approach to training deep networks that is robust to label noise. This approach introduces a non-linear processing layer (_noise model_) that models the statistics of the label noise into a convolutional neural network (CNN) architecture. The noise model and the CNN weights are learned jointly from noisy training data, which prevents the model from overfitting to erroneous labels. Through extensive experiments on several text classification datasets, we show that this approach enables the CNN to learn better sentence representations and is robust even to extreme label noise. We find that proper initialization and regularization of this noise model is critical. Further, by contrast to results focusing on large batch sizes for mitigating label noise for image classification, we find that altering the batch size does not have much effect on classification performance.

W. State University, Detroit MI 48202 Interactions Digital Roots, Ann Arbor MI 48104 {ishan.jindal, matthew.nokleby}@wayne.edu, {dpressel, blester}@interactions.com

## 1 Introduction

Deep Neural Networks (DNNs) have led to significant advances in the fields of computer vision He et al. (2016), speech processing Graves et al. (2013) and natural language processing Kim (2014); Young et al. (2018); Devlin et al. (2018). To be effective, supervised DNNs rely on large amounts of carefully labeled training data. However, it is not always realistic to assume that example labels are clean. Humans make mistakes and, depending on the complexity of the task, there may be disagreement even among expert labelers. Further, samples drawn from the class conditional densities with overlapping supports gives rise to the label noise in training datasets. To support noisy labels in data, we need new training methods that can be used to train DNNs directly from the corrupted labels to significantly reduce human labeling efforts. Zhu and Wu (2004) perform an extensive study on the effect of label noise on classification performance of a classifier and find that noise in input features is less important than noise in training labels.

In this work, we add a _noise model_ layer on top of our target model to account for label noise in the training set, following Jindal et al. (2016); Sukhbaatar et al. (2014). We provide extensive experiments on several text classification datasets with artificially injected label noise. We study the effect of two different types of label noise; _Uniform label flipping (Uni)_, where a clean label is swapped with another label sampled uniformly at random; and _Random label flipping (Rand)_ where a clean label is swapped with another label from the given number of labels sampled randomly over a unit simplex.

We also study the effect of different initialization, regularization, and batch sizes when training with noisy labels. We observe that proper initialization and regularization helps the noise model learn to be robust to even extreme amounts of noise. Finally, we use low-dimensional projections of the features of the training examples to understand the effectiveness of the noise model.

The rest of the paper is organized as follows. Section 2 discusses the various approaches in literature to handle label noise. In Section 3, we describe the problem statement along with the proposed approach. We describe the experimental setup and datasets in Section 4. We empirically evaluate the performance of the proposed approach along with the discussion in Section 5 and finally conclude our work in Section 6.



Related Work

Learning from label noise is a widely studied problem in the classical machine learning setting. Earlier works (Brodley and Friedl, 1999; Rebbapragada and Brodley, 2007; Manwani and Sastry, 2013) consider learning from noisy labels for a wide range of classifiers including SVMs (Natarajan et al., 2013) and fisher discriminants (Lawrence, 2001). Traditional approaches handle label noise by detecting and eliminating the corrupted labels. More details about these approaches can be found in (Frenay and Verleysen, 2014).

Recently, DNNs have made huge gains in performance over traditional methods on large datasets with very clean labels. However large real-world datasets often contain label errors. A number of works have attempted to address this problem of learning from corrupted labels for DNNs. These approaches can be divided into two categories; attempts to mitigate the effect of label noise using auxiliary clean data, and attempts to learn directly from the noisy labels.

**Presence of auxiliary clean data:** This line of research exploits a small, clean dataset to correct the corrupted labels. For instance, Li et al. (2017) learn a teacher network with clean data to re-weight a noisy label with a soft label in the loss function. Similarly, Veit et al. (2017) use the clean data as a label correction network. One can use this auxiliary source of information to do inference over latent clean labels (Vahdat, 2017). Further, Yao et al. (2018) models the auxiliary trustworthiness of noisy image labels to alleviate the effect of label noise. Though these methods show very promising results, the absence of clean data in some situations might hinder the applicability of these methods.

**Learning directly from noisy labels:** This research directly learns from the noisy labels by designing a robust loss function, or by modeling the latent labels. For instance, Reed et al. (2014), apply bootstrapping to the loss function to have consistent label prediction for similar images. Similarly, Joulin et al. (2016) alleviate the label noise effect by adequately weighting the loss function using the sample number. Jiang et al. (2017) propose a sequential meta-learning model that takes in a sequence of loss values and outputs the weights for the labels. Ghosh et al. (2017) further explores the conditions on loss functions such that the loss function is noise tolerant.

A number of approaches learn the transition from latent labels to the noisy labels. For example, Mnih and Hinton (2012) propose a noise adaptation framework for symmetric label noise. Based on this work, several other works (Sukhbaatar et al., 2014; Jindal et al., 2016; Patrini et al., 2017; Han et al., 2018) account for the label noise by learning a noisy layer on top of a DNN where the learned transition matrix represents the label flip probabilities. Similarly, Xiao et al. (2015) propose a probabilistic image conditioned noise model. Azadi et al. (2015) proposed an image regularization technique to detect and discard the noisy labeled images. Other approaches include building two parallel classifiers (Misra et al., 2016) where one classifier deals with image recognition and the other classifier models humans reporting bias.

All of these approaches have targeted image classification. In this work, we propose a framework for learning from noisy labels for text classification using a DNN architecture. Similar to (Sukhbaatar et al., 2014; Jindal et al., 2016; Patrini et al., 2017), we append a non-linear processing layer on top of this architecture to model the label noise. This layer helps the base architecture to learn better representations, even in the presence of label noise. We empirically show that, for better classification performance, the knowledge of noise transition matrix is not needed. Instead, the process forces the DNN to learn better sentence representations.

## 3 Problem Statement

In a supervised text classification setting where \({\bf x}_{i}\in{\mathbf{R}^{d}}\) is a \(d\)-dimensional word embedding of the \(i\)th word in a sentence of length \(l\) (padded wherever necessary), we represent the sample as a temporal embedding matrix \({\bf X}\in{\mathbf{R}^{d\times l}}\) which belongs to one of the \(K\) classes. Let the noise-free training set be denoted by

\[\mathcal{D}=\{({\bf X}_{1},y_{1}),({\bf X}_{2},y_{2}),\cdots,({\bf X}_{n},y_{ n})\},\]

where \(y_{i}\in\{1,\ldots,K\}\) represents the category of \(i\)th sample, \(n\) is the total number of training samples, and there is an unknown joint distribution \(p({\bf X},y)\) on the sample/label pairs. This temporal representation of a sample \({\bf X}\) is fed as input to a classifier on the training set \(\mathcal{D}\) with sample categories \(y\). However, as mentioned in Section 2, we cannot access the true noise-free samples labels and instead, observe noisy labels corrupted 

by an unknown noise distribution. Let this _noisy_ training set be denoted by

\[{\cal D}^{\prime}=\{({\bf X}_{1},y_{1}^{\prime}),({\bf X}_{2},y_{2}^{\prime}), \cdots,({\bf X}_{n},y_{n}^{\prime})\},\]

where \(y_{i}^{\prime}\) represents the corrupted label for the sentence \({\bf X}_{i}\). In this work, we suppose the label noise is _class-conditional_, where the noisy label \(y_{i}^{\prime}\) depends only on the true label \(y_{i}\), but not on the input \({\bf X}_{i}\) or any other labels \(y_{j}\) or \(y_{j}^{\prime}\). Under this model, the label noise is characterized by the conditional distribution

\[p(y^{\prime}=i|y=j)=\Phi_{ij},\]

which we describe via the \(K\times K\) column-stochastic matrix \(\Phi_{ij}\), parameterized by a matrix \({\bf Q}=\{\Phi_{ij}\}\).

In our experiments, we artificially inject label noise into the training and validation sets. We fix the noise distribution \(\Phi_{ij}\) and, for a training sample, we generate a noisy label by drawing i.i.d from this noise distribution \(\Phi_{ij}\). However, we do not alter the test labels.

Though the proposed approach works for any noise distribution, for this study, we focus on two different types of label flip distributions. We use a noise model parameterized by the overall probability of a label error, denoted by \(0\leq p\leq 1\). For a noise level \(p\), we set the noise distribution matrix

\[\Phi=(1-p){\bf I}+\frac{p}{K}{\bf II}^{K}, \tag{1}\]

and we call it _Uniform label flip noise model_. Here, **I** represents the identity matrix and **II** denotes the all-ones matrix. Similarly, we describe the _random label flip noise model_ as

\[\Phi=(1-p){\bf I}+p\Delta, \tag{2}\]

where **I** is the identity matrix, and \(\Delta\) is a matrix with zeros along the diagonal and remaining entries of each column are drawn uniformly and independently from the \(K-1\)-dimensional unit simplex. The label error probability for each class is \(p\), while the probability distribution _within_ the erroneous classes is drawn uniformly at random.

Our objective is to train a classifier on the noisy labeled sample categories on the training set \({\cal D}^{\prime}\) such that it jointly makes accurate predictions of the true label \(y\) and learns the noise transition matrix simultaneously, given \({\bf X}\). For the noisy dataset \({\cal D}^{\prime}\), it is straightforward to train a classifier that predicts the _noisy_ labels using conditional distribution for the noisy labeled input sentence \({\bf X}\):

\[p(y^{\prime}=\hat{y}^{\prime}|{\bf X})=\sum_{i}\Big{(}p(y^{ \prime}=\hat{y}^{\prime}|y=\hat{y}_{i})\\ p(y=\hat{y}_{i}|{\bf X})\Big{)}. \tag{3}\]

One can learn the classifier associated with \(p(y^{\prime}=\hat{y}^{\prime}|x)\) via standard training on the noisy set \({\cal D}^{\prime}\). To predict the clean labels by learning the conditional distribution \(p(y=\hat{y}_{i}|x)\) requires more effort, as we cannot extract the "clean" classifier from the noisy classifier when the label noise distribution is unknown.

### Proposed Framework

We refer to the DNN model without the final layer as the _base model_ or network _without noise model (WoNM)_. This model, along with the non-linear layer, is trained via back-propagation on the noisy training dataset. The non-linear processing layer in the _noise model_ transforms the base model outputs to match the noisy labels during the forward pass better and presents the denoised labels to the base model during the backward pass. The noise layer is parameterized by a square matrix \(\Psi\in{\bf R}^{K\times K}\)). At test time, we remove this learned noise model and use the output of the base model as final predictions.

We refer to the base model parameters as \(\Theta\). The base model outputs a probability distribution over the number of \(K\) categories denoted as \(p\left(y=\hat{y}_{i}|{\bf X};\Theta\right)\forall i\in\{1,2,\cdots,K\}\). During the forward pass the noise model transforms this output to obtain the noisy labels as

\[p\left(y^{\prime}|{\bf X};\Theta,\Psi\right)=\sigma\left(\Psi\times p\left(y|{ \bf X};\Theta\right)\right), \tag{4}\]

where \(\sigma(\cdot)\) represents the usual softmax operator. Note that both the equations (3) and (4) compute the probability distribution over noisy labels - our noise model does not learn a noise transition matrix. However, we assert that the knowledge of exact noise statistics is neither necessary nor sufficient for the better prediction results.

We learn the base model parameters \(\Theta\) and the noise model parameters \(\Psi\) by maximizing the log likelihood (4) over all of the training samples, 

minimizing the _cross-entropy_ loss:

\[{\mathcal{L}}(\Theta,\Psi;{\mathcal{D^{\prime}}})=-\frac{1}{n}\sum_{i=1 }^{n}\log\left[p\left(y^{\prime}|\mathbf{X}_{i};\Theta,\Psi\right)\right]\\ =-\frac{1}{n}\sum_{i=1}^{n}\log\left[\sigma\left(\Psi\times p\left( y|\mathbf{X}_{i};\Theta\right)\right)\right]_{y_{i}} \tag{5}\]

Similar to (Sukhbaatar et al., 2014), we initialize the noise model weights to the identity matrix. Since DNNs have high capacity, we may encounter the situation when the base model itself absorbs all the label noise and, thus, the noise model does not learn anything at all. In order to avoid this situation, and to prevent overfitting, we apply \(l_{2}\) regularization to the noise model. However, we want the noise model to overfit the label noise. In the experiment section, we observe that with proper regularization and weight initialization the noise model absorbs most of the label noise. Finally, we train the entire network according to the following loss function:

\[{\mathcal{L}}=-\frac{1}{n}\sum_{i=1}^{n}\log\left[\sigma\left( \Psi\times p\left(y|\mathbf{X}_{i};\Theta\right)\right)\right]_{y_{i}}\\ +\frac{1}{2}\lambda||\Psi||_{2}^{2}. \tag{6}\]

Here, \(\lambda\) is a tuning parameter and we validate the value of \(\lambda\) by repeating the experiment multiple times with multiple \(\lambda\) values over different datasets and choose the one with better classification performance on the validation set for the respective datasets. A value of \(\lambda=0.01\) works best.

## 4 Datasets and Experimental Setup

In this section, we empirically evaluate the performance of the proposed approach for text classification and compare our results with the other methods.

### General Setting

In all the experiments, we use a publicly-available deep learning library _Baseline_ - a fast model development tool for NLP tasks (Pressel et al., 2018). For all the different datasets, we choose a commonly-used, high-performance model from (Kim, 2014) as a base model. To examine the robustness of the proposed approach, we intentionally flip the class labels with \(0\%\) to \(70\%\) label noise, in other words: \(p\in\{0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7\}\), and observe the effect of different types of label flipping, such as uniform (_Uni_) and random (_Rand_) label flipping, along with instance-dependent label noise. For all the experiments, we use early stopping based on validation set accuracy where the class labels in validation are also corrupted.

We indicate the performance of a standard deep network _Without Noise model (WoNM)_ on the noisy label dataset. We also plot the results for the stacked _Noise Model Without Regularization (NMwRegu)_ and stacked _Noise Model With Regularization (NMwRegu)_. Unless otherwise stated, in all the deep networks with the stacked noise model, we initialize the noise layer parameters as an identity matrix. We further analyze the effect of the noise layer initialization on the overall performance. We define _TDwRegu_ as the stacked noise model with regularization, initialized with true injected noise distribution and _Randwkegu_ as the stacked noise model with regularization, initialized randomly. We run all experiments five times and report the mean accuracy.

### Datasets

Here, we describe all the text classification datasets used to evaluate the performance of the proposed approach. The base model architecture is the same for all datasets. For each set, we tune the number of filter windows and filter lengths using the development set. Along with the description, we also provide the hyper-parameters we selected for each. Table 1 summarizes the basic statistic of the datasets.

1. SST-21(Socher et al., 2011): Stanford Sentiment Treebank dataset for predicting the sentiment of movie reviews. The classification task involves detecting positive or negative reviews. Using the base model with

\begin{table}
\begin{tabular}{c l||c|c|c|c|c} \hline  & Dataset & K & L & N & T & Type \\ \cline{2-7}  & SST-2 & 2 & 19 & 76961 & 1821 & Balanced \\  & TREC & 6 & 10 & 5000 & 500 & Not Balanced \\  & AG-News & 4 & 38 & 110K & 10K & Balanced \\  & DBpedia & 14 & 29 & 504K & 70K & Balanced \\ \hline \end{tabular}
\end{table}
Table 1: Summary of text classification datasets; K: denotes the number of classes, L: represents the average length of sentence, N: denotes the number of training samples, T: represents the number of test samples, Type: describes whether the dataset is balanced.

 

clean labels we obtain classification accuracy of \(87.27\%\). For this dataset, the base model network architecture consists of an input and embedding layer + \([3,4,5]\) feature windows with 100 feature maps each and dropout rate \(0.5\) with batch size 50.
2. TREC2 (Voorhees and Tice, 1999): A question classification dataset consisting of fact based questions divided into broad semantic categories. We use a six-class version of TREC dataset. For this dataset, the base model network architecture consists of an input and embedding layer + \([3]\) one feature windows with 100 feature maps and dropout rate \(0.5\) with batch size 10. Footnote 2: [http://cogcomp.cs.illinois.edu/Data/QA/QC/](http://cogcomp.cs.illinois.edu/Data/QA/QC/)
3. Ag-News3 (Zhang et al., 2015): A large-scale, four-class topic classification dataset. It contains approx 110K training samples. For this dataset, the base model network architecture consists of Input layer + Embedding layer + \([3,4,5]\) feature windows with 200 feature maps and dropout rate \(0.5\) with batch size 100. Footnote 3: [http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)
4. DBpedia3 (Zhang et al., 2015): A large scale 14-class topic classification dataset containing \(36K\) training samples per category. For this dataset, the base model network architecture consists of Input layer + Embedding layer + \([1,2,3,4,5,7]\) feature windows with 400 feature maps each and dropout rate \(0.5\) with batch size 1024.

\begin{table} width=center 

\end{table}
Table 2: Test performance for different text classification datasets 

For all the datasets, we use Rectified Linear Units (ReLU) and fix the base model architecture. We use early stopping on dev. sets for all the datasets. We run all the experiments 5 times and report the average classification accuracy in Table 2. We train all the networks end-to-end via stochastic gradient descent over shuffled mini-batches with the Adadelta update rule Zeiler (2012) except for the DBpedia, where we use SGD. In order to improve base model performance, we initialize the word embedding layer with the publicly available _word2vec_ word vectors Mikolov et al. (2013) for all the datasets except for DBpedia, where we use _GloVe_ embeddings Pennington et al. (2014).

## 5 Results and Discussion

We evaluate the performance of our model in Table 2 for each datasets in the presence of uniform and random label noise and compare the performance with the base model (_WoNM_) as our baseline. For all datasets, the proposed approach is significantly better than the baseline for both random and uniform label noise. For all datasets, we observe a gain of approximately \(30\%\) w.r.t the baseline in the presence of extreme label noise. We do observe a drop in classification accuracy as we increase the percentage of label noise but even at the extreme label noise our method outperformed the baseline method. Interestingly, if we assume an oracle to determine prior knowledge of true noise distribution (_TDwRegu01_), it does not necessarily improve classification performance, especially for multi-class classification problems. For binary classification, using the SST-2 dataset, we did observe that the noise model initialized with the true noise distribution works better than all the other models. In addition to this, we also observe a slight performance gain for the proposed approach over the baseline with clean labels - perhaps due to label noise inherent in the datasets.

### Effect of Different Regularizers

The _NMwRegu01_ performs better in all cases for both types of label noise. We plot the weight matrix learned by all the noise models in all the noise regimes. For brevity, we only plot the weight matrix for AG-News datasets with \(30\%\) label noise in Fig.1. We find that \(l_{2}\) regularization diffuses the diagonal weight elements and learns more smoothed off-diagonal elements which resemble the corresponding input label noise distribution in Fig. 0(d). This also means that, without regularization, the noise model has less ability to diffuse the diagonal elements which leads to poor classification performance. Therefore, we use a regularizer (\(l_{2}\)) to diffuse the diagonal entries.

In some cases, especially for low label noise, we find the \(l_{2}\) regularization with a small penalty works better than a large penalty since, for low label noise, learning a less diffuse noise is beneficial. The proposed approach scales to a large number of label categories, as evident from the experiments on DBpedia dataset in the last row of Table .2.

Figure 1: AG-News Dataset: a) Input random label noise; (b-f) learned weight matrix learned by different noise models.

Figure 2: Effect of noise model initialization scaling on the classification performance





### Effect of Different Scaling Factors on Noise Layer Initialization

We initialize the noise model weights as identity matrices with gain equal to the number of classes (gain = \(K\)) for all experiments. We observe the effect of different gain values on the overall performance of the proposed network in Fig. 2 where on x-axis we vary the scaling factor - a function of number of classes in the dataset. We plot the classification performance for the DBpedia dataset with \(50\%\) random noise. For each noise model in Fig. 1(a), we find that setting the gain to \(K\) works best and any other gain results in poor performance.

In Fig. 1(b) we plot the Frobenius norm of the learned noise model weights with respect to the different gain values. We find that, using the high gain initialization, the model learns a high noise model norm, resulting in poor classification performance. This finding provides support to the claim in Liao et al. (2018) that "higher layer norm leads to highest test errors."

### Effect of Batch Size

We also observe the effect of different batch sizes on performance as described in Rolnick et al. (2017). For all datasets, we do observe small performance gains for highly non-uniform noisy labels, for instance \(70\%\), in Fig. 3 row 2. However, for uniform label flips, we do not observe performance gains with increasing batch size.

### Instance Dependent Label Noise

We further investigate the performance of the proposed approach on instance-dependent label noise by flipping each class labels with different noise percentages as shown in Fig. 4. For brevity, we present results on AG-News dataset in Fig. 4. On this type of label noise, the performance of the proposed approach is far better than the baseline with a performance improvement of \(\sim 6\%\). The noise model learned by the proposed approach is shown in Fig. 3(b) and we show the normalized weight matrix in Fig. 3(c). We observe that the learned noise model is able to capture the input label noise statistics and is highly correlated to the input noise distribution with Pearson Correlation Coefficient \(0.988\).

### Understanding Noise Model

In order to further understand the noise model, we first train the base model and the proposed model on noisy labels. Afterward, we collect

Figure 3: Effect of batch size on label noise classification for different datasets. [Best viewed in color]

the last fully-connected layer's activations for all the training samples and treat them as the learned feature representation of the input sentence. We get two different sets of feature representations, one corresponding to the base model (_TRB_), and the other corresponding to the proposed model (_TRPr_). Given these learned feature representations - the artificially injected noisy labels and the true labels of the training data - we learn two different SVMs for each model, with and without noise. For the base model, for both SVMs, we use TRB representation as inputs and train the first SVM with the true labels as targets and the second SVM with the unreliable labels as targets. Simi

Figure 4: AG-News Dataset: a) input instance dependent label noise; b) learned weight matrix by proposed approach; c) column normalization of (b).

Figure 5: t-SNE visualization of the last layer activations of a base network before softmax for TREC Dataset with \(50\%\) corrupted labels; First row in (a) when the corresponding true labels are superimposed on the t-SNE data points; Second row in (a) when the noisy labels are superimposed onto the t-SNE data points. [Best viewed in color]



larly, we train two SVMs for the proposed model. After training, we evaluate the performance of all the learned SVMs on clean test data in Table 3, where the 1st column represents the corresponding model performance, "Noisy" and "True" column represents the SVM performance when trained on noisy and clean labels, respectively. We run these experiments for different datasets with different label noise.

The SVM, trained on TRB and noisy labels, is very close to the base model performance (3). This suggests that the base model is just fitting the noisy labels. On the other hand, when we train an SVM on the TRPr representations with true labels as targets, the SVM achieves the proposed model performance. This means that the proposed approach helps the base model to learn better feature representations even with the noisy targets, which suggest that this noise model is learning a label denoising operator.

We analyze the representation of training samples in feature domain by plotting the t-SNE embeddings [20] of the IRB and TRPr. For brevity, we plot the t-SNE visualizations for TREC dataset with \(50\%\) label noise in Fig. 5 .

For each network, we show two different t-SNE plots. For example in Fig. 4(a) we plot two rows of t-SNE embeddings for the proposed model. In the first row of Fig. 4(a), each training sample is represented by its corresponding true label, while in the second row (the noisy label plot) each training sample is represented by its corresponding noisy label. We observe that, as the learning process progresses, the noise model helps the base model to cluster the training samples in the feature domain. With each iteration, we can see the formation of clusters in Row 1. However, in Row 2, when the noisy labels are superimposed, the clusters are not well separated. This means that the noise model denoises the labels and presents the true labels to the base network to learn.

In Fig. 4(b), we plot two rows of t-SNE embeddings of the TRB representations. It seems that the network directly learns the noisy labels. This provides further evidence to support [76]'s finding that the deep network memorizes data without knowing of true labels. In Row 2 of Fig. 4(b), we can observe that the network learns noisy features representations which can be well clustered according to given noisy labels.

## 6 Conclusion

In this work, we propose a framework to enable a DNN to learn better sentence representations in the presence of label noise for text classification tasks. To model the label noise, we append a non-linear noise model on top of the standard DNN architecture. With proper initialization and regularization, the noise model is able to absorb most of the label noise and helps the base model to learn better sentence representations.

## Acknowledgments

We thank the anonymous reviewers for their detailed and insightful comments. We would also like to thank Patrick Haffner, Sagnik Ray Choudhury, Yanjie Zhao and Amy Hemmeter for their valuable discussions with us during the course of this research.

## References

* Azadi et al. (2015) Samaneh Azadi, Jiashi Feng, Stefanie Jegelka, and Trevor Darrell. 2015. Auxiliary image regularization for deep cnns with noisy labels. _arXiv preprint arXiv:1511.07069_.
* Brodley and Friedl (1999) Carla E Brodley and Mark A Friedl. 1999. Identifying mislabeled training data. _Journal of artificial intelligence research_, 11:131-167.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Frenay and Verleysen (2014) Benoit Frenay and Michel Verleysen. 2014. Classification in the presence of label noise: a survey. _IEEE transactions on neural networks and learning systems_, 25(5):845-869.
* Ghosh et al. (2017) Aritra Ghosh, Himanshu Kumar, and PS Sastry. 2017. Robust loss functions under label noise for deep neural networks. In _AAAI_, pages 1919-1925.
* Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In _Acoustics, speech and signal processing (icassp), 2013 ieee international conference on_, pages 6645-6649. IEEE.
* Han et al. (2018) Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. 2018. Masking: A new perspective of noisy supervision. _arXiv preprint arXiv:1805.08193_.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778 

* Jiang et al. (2017) Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2017. Mentornet: Regularizing very deep neural networks on corrupted labels. _arXiv preprint arXiv:1712.05055_.
* Jindal et al. (2016) Ishan Jindal, Matthew Nokleby, and Xuewen Chen. 2016. Learning deep networks from noisy labels with dropout regularization. In _Data Mining (ICDM), 2016 IEEE 16th International Conference on_, pages 967-972. IEEE.
* Joulin et al. (2016) Armand Joulin, Laurens van der Maaten, Allan Jagri, and Nicolas Vasilache. 2016. Learning visual features from large weakly supervised data. In _European Conference on Computer Vision_, pages 67-84. Springer.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751.
* Lawrence (2001) Neil D Lawrence. 2001. Estimating a kernel fisher discriminant in the presence of label noise. In _ICML_. Citeseer.
* Li et al. (2017) Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. 2017. Learning from noisy labels with distillation. In _ICCV_, pages 1928-1936.
* Liao et al. (2018) Qianli Liao, Brando Miranda, Andrzej Banburski, Jack Hidary, and Tomaso Poggio. 2018. A surprising linear relationship predicts test performance in deep networks. _arXiv preprint arXiv:1807.09659_.
* Manwani and Sastry (2013) Naresh Manwani and PS Sastry. 2013. Noise tolerance under risk minimization. _IEEE transactions on cybernetics_, 43(3):1146-1151.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* Misra et al. (2016) Ishan Misra, C Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. 2016. Seeing through the human reporting bias: Visual classifiers from noisy homnentrentic labels. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 2930-2939.
* Mnih and Hinton (2012) Volodymyr Mnih and Geoffrey E Hinton. 2012. Learning to label aerial images from noisy data. In _Proceedings of the 29th International conference on machine learning (ICML-12)_, pages 567-574.
* Natarajan et al. (2013) Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuji Tewari. 2013. Learning with noisy labels. In _Advances in neural information processing systems_, pages 1196-1204.
* Patrini et al. (2017) Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. 2017. Making deep neural networks robust to label noise: A loss correction approach. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 1944-1952.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Pressel et al. (2018) Daniel Pressel, Sangnik Ray Choudhury, Brian Lester, Yanjie Zhao, and Matt Barta. 2018. Baseline: A library for rapid modeling, experimentation and development of deep learning algorithms targeting nlp. In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_, pages 34-40. Association for Computational Linguistics.
* Rebbapragada and Brodley (2007) Umaa Rebbapragada and Carla E Brodley. 2007. Class noise mitigation through instance weighting. In _European Conference on Machine Learning_, pages 708-715. Springer.
* Reed et al. (2014) Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. 2014. Training deep neural networks on noisy labels with bootstrapping. _arXiv preprint arXiv:1412.6596_.
* Rolnick et al. (2017) David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. 2017. Deep learning is robust to massive label noise. _arXiv preprint arXiv:1705.10694_.
* Socher et al. (2011) Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In _Proceedings of the conference on empirical methods in natural language processing_, pages 151-161. Association for Computational Linguistics.
* Sukhbaatar et al. (2014) Sainbayar Sukhbbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. 2014. Training convolutional networks with noisy labels. _arXiv preprint arXiv:1406.2080_.
* Vahdat (2017) Arash Vahdat. 2017. Toward robustness against label noise in training deep discriminative neural networks. In _Advances in Neural Information Processing Systems_, pages 5596-5605.
* Van Der Maaten (2014) Laurens Van Der Maaten. 2014. Accelerating t-sne using tree-based algorithms. _The Journal of Machine Learning Research_, 15(1):3221-3245.
* Veit et al. (2017) Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge J Belongie. 2017. Learning from noisy large-scale datasets with minimal supervision. In _CVPR_, pages 6575-6583.
* Voorhees and Mice (1999) Ellen M Voorhees and Dawn M Tice. 1999. The trec-8 question answering track evaluation. In _TREC_, volume 1999, page 82.

 

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. 2015. Learning from massive noisy labeled data for image classification. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 2691-2699.
* Yao et al. (2018) Jiangchao Yao, Jiajie Wang, Ivor W Tsang, Ya Zhang, Jun Sun, Chengqi Zhang, and Rui Zhang. 2018. Deep learning from noisy image labels with quality embedding. _IEEE Transactions on Image Processing_.
* Young et al. (2018) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018. Recent trends in deep learning based natural language processing. _IEEE Computational Intelligence Magazine_, 13(3):55-75.
* Zeiler (2012) Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. _arXiv preprint arXiv:1212.5701_.
* Zhang et al. (2016) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016. Understanding deep learning requires rethinking generalization. _arXiv preprint arXiv:1611.03530_.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Advances in neural information processing systems_, pages 649-657.
* Zhu and Wu (2004) Xingquan Zhu and Xindong Wu. 2004. Class noise vs. attribute noise: A quantitative study. _Artificial Intelligence Review_, 22(3):177-210.