We propose several simple approaches to training deep neural networks on data with noisy labels. We introduce an extra noise layer into the network which adapts the network outputs to match the noisy label distribution. The parameters of this noise layer can be estimated as part of the training process and involve simple modifications to current training infrastructures for deep networks. We demonstrate the approaches on several datasets, including large scale experiments on the ImageNet classification benchmark, showing how additional noisy data can improve state-of-the-art recognition models. 1 Introduction In recent years, deep learning methods have shown impressive results on image classification tasks. However, this achievement is only possible because of large amount of labeled images. Labeling images by hand is a laborious task and takes a lot of time and money. An alternative approach is to generate labels automatically. This includes user tags from social web sites and keywords from image search engines. Considering the abundance of such noisy labels, it is important to find a way to utilize them in deep learning. Unfortunately, those labels are very noisy and unlikely to help training deep networks without additional tricks. Our goal is to study the effect label noise on deep networks, and explore simple ways of improvement. We focus on the robustness of deep networks instead of data cleaning methods, which are well studied and can be used together with robust models directly. Although many noise robust classifiers are proposed so far, there are not many works on training deep networks on noisy labeled data, especially on large scale datasets. Our contribution in this paper is a novel way of modifying deep learning models so they can be effectively trained on data with high level of label noise. The modification is simply done by adding a linear layer on top of the softmax layer, which makes it easy to implement. This additional layer changes the output from the network to give better match to the noisy labels. Also, it is possible to learn the noise distribution directly from the noisy data. Using real-world image classification tasks, we demonstrate that the model actually works very well in practice. We even show that random images without labels (complete noise) can improve the classification performance. 2 Related Work In any classification model, degradation of performance is inevitable when there is noise in training labels [13, 15]. A simple approach to handle noisy labels is a data preprocessing stage, where labels suspected to be incorrect are removed or corrected [1, 3]. However, a weakness of this approach is the difficulty of distinguishing informative hard samples from harmful mislabeled ones [6]. Instead, in this paper, we focus on models robust to presence of label noise. 1 ar X iv :1 40 6. 20 80 v1 [ cs .C V ] 9 J un 2 01 4 The effect of label noise is well studied in common classifiers (e.g., SVMs, kNN, logistic regression), and their label noise robust variants have been proposed. See [5] for comprehensive review. A more recent work [2] proposed a generic unbiased estimator for binary classification with noisy labels. They employ a surrogate cost function that can be expressed by a weighted sum of the original cost functions, and gave theoretical bounds on the performance. In this paper, we will also consider this idea and extend it multiclass. A cost function similar to ours is proposed in [2] to make logistic regression robust to label noise. They also proposed a learning algorithm for noise parameters. However, we consider deep networks, a more powerful and complex classifier than logistic regression, and propose a different learning algorithm for noise parameters that is more suited for back-propagation training. Considering the recent success of deep learning [8, 17, 16], there are very few works about deep learning from noisy labels. In [11, 9], noise modeling is incorporated to neural network in the same way as our proposed model. However, only binary classification is considered in [11], and [9] assumed symmetric label noise (noise is independent of the true label). Therefore, there is only a single noise parameter, which can be tuned by cross-validation. In this paper, we consider multiclass classification and assume more realistic asymmetric label noise, which makes it impossible to use cross-validation to adjust noise parameters (there can be a million parameters). 3 Approach In this paper, we consider two approaches to make an existing classification model, which we call the base model, robust against noisy labels: bottom-up and top-down noise models. In the bottomup model, we add an additional layer to the model that changes the label probabilities output by the base model so it would better match to noisy labels. Top-down model, on other hand, changes given noisy labels before feeding them to the base model. Both models require a noise model for training, so we will give an easy way to estimate noise levels using clean data. Also, it is possible to learn noise distribution from noisy data in the bottom-up model. Although only deep neural networks are used in our experiments, the both approaches can be applied to any classification model with a cross entropy cost. 3.1 Bottom-up Noise Model We assume that label noise is random conditioned on the true class, but independent of the input x (see [10] for more detail about this type of noise). Based on this assumption, we add an additional layer to a deep network (see Figure 1) that changes its output so it would better match to the noisy labels. The weights of this layer corresponds to the probabilities of a certain class being mislabeled to another class. Because those probabilities are often unknown, we will show how estimate them from additional clean data, or from the noisy data itself. Let D be the true data distribution generating correctly labeled samples (x, y∗), where x is an input vector and y∗ is the corresponding label. However, we only observe noisy labeled samples (x, ỹ) that generated from a some noisy distribution D̃. We assume that the label noise is random conditioned on the true labels. Then, the noise distribution can be parameterized by a matrix Q = {qji}: qji := p(ỹ = j|y∗ = i). Q is a probability matrix because its elements are positive and each column sums to one. The probability of input x being labeled as j in D̃ is given by p(ỹ = j|x, θ) = ∑ i p(ỹ = j|y∗ = i)p(y∗ = i|x) = ∑ i qjip(y ∗ = i|x, θ). (1) where p(y∗ = i|x, θ) is the probabilistic output of the base model with parameters θ. If the true noise distribution is known, we can modify this for noisy labeled data. During training, Q will act as an adapter that transforms the model’s output to better match the noisy labels. Deep	  network Learnin from noisy labels in deep neural networks Sainbayar Sukhbaatar Dept. of Computer Science, NYU, 715 Broadway, New Y rk, NY 10003 sainbar@cs.nyu.edu Rob Fergus Courant Institute, NYU, 715 Broadway, New York, NY 10003 fergus@cs.nyu.edu
Output: Learning from Noisy Labels with Deep Neural Networks