Trimmed categorical cross‐entropy for deep learning with label noise

Electronics LettersVolume 55, Issue 6 p. 319-320 Image and vision processing and display technologyFree Access Trimmed categorical cross-entropy for deep learning with label noise A. Rusiecki, Corresponding Author A. Rusiecki andrzej.rusiecki@pwr.edu.pl Department of Computer Engineering, Wroclaw University of Science and Technology, Wrocław, PolandSearch for more papers by this author A. Rusiecki, Corresponding Author A. Rusiecki andrzej.rusiecki@pwr.edu.pl Department of Computer Engineering, Wroclaw University of Science and Technology, Wrocław, PolandSearch for more papers by this author First published: 01 March 2019 https://doi.org/10.1049/el.2018.7980Citations: 31AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Deep learning methods are nowadays considered as state-of-the-art approach in many sophisticated problems, such as computer vision, speech understanding or natural language processing. However, their performance relies on the quality of large annotated datasets. If the data are not well-annotated and label noise occur, such data-driven models become less reliable. In this Letter, the authors present very simple way to make the training process robust to noisy labels. Without changing network architecture and learning algorithm, the authors apply modified error measure that improves network generalisation when trained with label noise. Preliminary results obtained for deep convolutional neural networks, trained with novel trimmed categorical cross-entropy loss function, revealed its improved robustness for several levels of label noise. Introduction As deep neural networks are usually trained on large supervised datasets, they can potentially build models representing high-level abstractions [1, 2]. Such networks became popular mainly due to their top performance in many real-life problems. However, these impressive results could be obtained only with well-annotated huge data collections. Some of the large datasets are publicly available [3], but very often one needs to prepare training data manually, which can be expensive and time-consuming process. Even if the data labels are generated based on data mining algorithms, or web search engines, they still suffer from label noise. In this Letter, we propose a simple way to make the training process more robust to noisy data. Without changing the network architecture or learning algorithm we apply modified loss as new error criterion. The loss is based on the idea of robust trimmed estimators applied to well-known categorical cross-entropy error. Related work Training neural networks in the presence of noise or outliers was very often considered for regression tasks [4-8]. Basic methods to make the network more robust to outlying data points usually involve modified error functions. New loss measures decrease the influence that large residuals may have on the whole training process. In instance selection algorithms, such as compression methods and noise filters [9, 10], outliers can be detected and excluded from datasets. These methods can be potentially applied also to classification tasks. To address the problem of training deep models under label noise, several approaches were recently proposed. Some of the methods are designed to learning directly from data containing incorrect labels [11-15], while the others focus on cleaning training patterns. In this case, improperly labelled examples can be corrected, or simply removed from training sets. The noise can be modelled as conditionally independent from the input [16, 17], or considered as image-conditional label noise [18, 19]. In this Letter, we present novel approach based on modified loss function. Corrected error measures were also considered in [20, 21]. Modified loss Our modified trimmed categorical cross-entropy (TCCE) loss uses an idea of trimmed robust estimators. These estimators can be also used as the basis to define new error measures for neural network training [6, 8]. However, such efforts were made for regression problems, where robust equivalents of mean squared error (MSE) can be applied. In the case of classification, such approaches become less effective, mainly because they measure rather average deviation from targets, not classification accuracy. The most popular error measure in training deep structures on classification tasks is categorical (or binary, in the case of binary classification) cross-entropy (CCE). We consider a training set consisting of N pairs: , where denotes the -dimensional ith input vector, the corresponding C-dimensional network target, and its output. In other words, we want to classify -dimensional input to one of C categories. CCE loss can be then defined as (1)where is a binary indicator function that detects whether the ith training pattern belongs to cth category. The target can be interpreted as true, and output as predicted probability distribution for ith observation belonging to class c. Our idea is to substitute CCE loss with more robust error measure. As in least trimmed absolute value loss, defined in our previous work [8], we introduce trimming process to the calculation of network error. However, in this novel approach trimming is applied to CCE residuals. New TCCE loss is then defined as (2)where are ordered losses for each observation, written as (3)The trimming parameter h determines quantity of smallest residuals that are taken into account in calculating training loss. It can be set based on the expected amount of noisy labels in the training set. Potentially, one could try also to make h dependent on residuals calculated in certain phases of training. Experimental setup Our experiments were conducted on two well-known and representative datasets: MNIST [22] and CIFAR-10 [23]. We used network architectures similar to those described in [20], implemented in Python 3.6 with TesorFlow. For several levels of label noise, generalisation ability of MSE, CCE and two versions (with and ) of novel trimmed CCE loss were tested. Convolutional deep neural networks were trained with Adam algorithm [24] for 250 epochs, with parameters set to: learning rate , and . All accuracies were averaged over six runs of simulations performed on GTX 1080Ti GPU. The hidden layers architectures were as follows, for MNIST dataset: Convolution max pooling fully connected 1024 neurons fully connected 1024 neurons, and for CIFAR-10: 2 Convolutional layers max pooling 2 Convolutional layers max pooling fully connected 512 neurons. To make the influence of incorrect labels more prominent, we have not applied dropout in network training. The label noise was introduced following the uniform noise model. For each training pattern, with probability its label was changed into a label uniformly drawn from all other possible (and incorrect) labels. The probability was varied from 0 up to 60%. Simulation results Averaged test accuracies are presented in Figs. 1 and 2. The results obtained for MNIST dataset can be a little bit surprising for clean training data. All the tested methods achieved similar performance reaching up to 99% accuracy. The situation changes when label noise appears: for , TCCE with even improved its performance, while other approaches perform worse with TCCE with obtaining the lowest accuracy. When probability of incorrect labels increases, the latter loss strongly dominates the others, that perform still in the same order (starting from the most efficient): TCCE with , MSE and CCE. Fig 1Open in figure viewerPowerPoint Averaged test accuracy for MNIST dataset (input size 2828, 10 classes, 60k training and 10k test patterns) Fig 2Open in figure viewerPowerPoint Averaged test accuracy for CIFAR-10 dataset (input size , 10 classes, 50k training and 10k test patterns) If we consider more difficult CIFAR-10 dataset, and analyse Fig. 2, we can observe that for training data without incorrect labels, the best results were obtained for CCE and TCCE with . For increasing probability of noisy labels, TCCE with again outperforms other methods. However, when probability of noise ranges highest values, the best accuracies were surprisingly achieved by simple MSE loss. Such phenomenon may be caused by the fact that no dropout was used during network training. What may be observed in both Figs. 1 and 2 is that properly parametrised TCCE can indeed improve network robustness to incorrect labels. Further investigation is needed to design a way of automatic tuning trimming parameter h, based on current residuals. Conclusions In this Letter, we presented novel approach to training deep models under label noise. Our method is based on new error measure, TCCE, which can be easily applied in traditional learning algorithms. This approach does not require any changes to network architecture. Preliminary experimental results, conducted on popular classification datasets, revealed that TCCE can make the whole training process more robust to incorrect labels. Its generalisation abilities, when label noise occurs, are stronger than for traditional approaches. Properly parametrised TCCE performs well also for clean training dataset. Our future efforts should be then directed towards automatic selection of trimming parameter, which could be based on current training residuals. Acknowledgment This work has been supported by WUST internal grant 0401/0147/18. References 1Bengio, Y., Lamblin, P., Popovici, D. et. al.,: ‘ Greedy layer-wise training of deep networks’ NIPS 2006, Vancouver, BC, Canada, December 2006, pp. 153– 160 2Salakhutdinov, R., Hinton, G.E.: ‘ Semantic hashing’. Proc. of the 2007 Workshop on Information Retrieval and applications of Graphical Models (SIGIR 2007), 2007, Amsterdam 3Vahdat, A.: ‘ Toward robustness against label noise in training deep discriminative neural networks’. Neural Information Processing Systems (NIPS), Long Beach, CA, USA, December 2017 4Chen, D., Jain, R.: ‘A robust backpropagation learning algorithm for function approximation’, IEEE Trans. Neural Netw., 1994, 5, (3), pp. 467– 479 (https://doi/org/10.1109/72.286917) 5Chuang, C.C., Su, S.F., Hsiao, C.C.: ‘The annealing robust backpropagation (ARBP) learning algorithm’, IEEE Trans. Neural Netw., 2000, 11, (5), pp. 1067– 1077 (https://doi/org/10.1109/72.870040) 6El-Melegy, M., Essai, M., Ali, A.: ‘ Robust training of artificial feedforward neural networks, in A.E. Abraham, A. Vasilakos, A. Pedrycz, W. Hassanien (Eds.): ‘ Foundations of Computational, Intelligence Volume 1, Studies in Computational Intelligence’, vol. 201, Springer, Berlin, Heidelberg, 2009, pp. 217– 242 7Korodos, M., Rusiecki, A.: ‘Reducing noise impact on MLP training’, Soft Comput., 2016, 20, (1), pp. 49– 65 (https://doi/org/10.1007/s00500-015-1690-9) 8Rusiecki, A.: ‘Robust learning algorithm based on LTA estimator’, Neurocomputing, 2013, 120, pp. 624– 632 (https://doi/org/10.1016/j.neucom.2013.04.008) 9Salvador, G., Derrac, J., Ramon, C.: ‘Prototype selection for nearest neighbor classification: taxonomy and empirical study’, IEEE Trans. Pattern Anal. Mach. Intell., 2012, 34, pp. 417– 435 (https://doi/org/10.1109/TPAMI.2011.142) 10Wilson, D.L.: ‘Asymptotic properties of nearest neighbor rules using edited data’’, Trans. Syst. Man Cybern., 1972, SMC-2, (3), pp. 408– 421 (https://doi/org/10.1109/TSMC.1972.4309137) 11Guan, M.Y., Gulshan, V., Dai, A.M. et. al.,: ‘Who said what: modeling individual labelers improves classification’, 2017, arXiv:1703.08774 12Joulin, A., van der Maaten, L., Jabri, A. et. al.,: ‘ : learning visual features from large weakly supervised data’. European Conf. on Computer Vision, (ECCV), Amsterdam, Netherlands, October 2016 13Misra, I., Lawrence, Z.C., Mitchell, M. et. al.,: ‘ Seeing through the human reporting bias: visual classifiers from noisy human-centric labels’. Computer Vision and Pattern Recognition (CVPR), Las Vegas, CA, USA, 26 June–1 July 2016 14Reed, S., Lee, H., Anguelov, D. et. al.,: ‘Training deep neural networks on noisy labels with boot- strapping’, 2014, arXiv preprint arXiv:1412.6596 15Van Horn, G., Branson, S., Farrell, R. et. al.,: ‘ Building a bird recognition app and large scale dataset with citizen scientists: the fine print in fine-grained dataset collection’. Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, June 2015 16Natarajan, N., Inderjit, S.D., Ravikumar, P.K. et. al.,: ‘ Learning with noisy labels’. Advances in Neural Information Processing Systems (NIPS), Lake Tahoe, CA, USA, December 2013 17Sukhbaatar, S., Bruna, J., Paluri, M. et. al.,: ‘Training convolutional networks with noisy labels’, 2014, arXiv preprint arXiv:1406.2080 18Veit, A., Alldrin, N., Chechik, G. et. al.,: ‘ Learning from noisy large-scale datasets with minimal supervision’. Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, July 2017 19Xiao, T., Xia, T., Yang, Y. et. al.,: ‘ Learning from massive noisy labeled data for image classification’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, June 2015, pp. 2691– 2699 20Ghosh, A., Kumar, H., Sastry, P.S.: ‘Robust Loss Functions under Label Noise for Deep Neural Networks’, 2017, arXiv:1712.09482v1 21Patrini, G., Rozza, A., Menon, A. et. al.,: ‘ Making neural networks robust to label noise: a loss correction approach’. Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, July 2017 22LeCun, Y., Cortes, C.: ‘MNIST handwritten digit database’, Available at http://yann.lecun.com/exdb/mnist/ 23Krizhevsky, A.: ‘Learning multiple layers of features from tiny images’. Technical report, 2009 24Kingma, D., Ba, J.: ‘Adam: a method for stochastic optimization’, 2014, arXiv preprint arXiv:1412.6980 Citing Literature Volume55, Issue6March 2019Pages 319-320 FiguresReferencesRelatedInformation