Robust Learning via Golden Symmetric Loss of (un)Trusted Labels

1 INTRODUCTION . Diverse datasets collected from the public domain which power up deep learning models present new challenges – highly noisy labels . It is not only time consuming to collect labels but also difficult to ensure a consistent label quality due to various annotation errors ( Patrini et al. , 2017 ) and adversarial attacks ( Goodfellow et al. , 2015 ) . The large capacity of deep learning models enables effective learning from complex datasets but also suffers from overfitting to the noise structure in the dataset . The curse of memorization effect ( Jiang et al. , 2018 ) can degrade the accuracy of deep learning models in the presence of highly noisy labels . For example , in ( Zhang et al. , 2017 ) the accuracy of AlexNet to classify CIFAR10 images drops from 77 % to 10 % , when there are randomly flipped labels . Designing learning models that can robustly train on noisy labels is thus imperative . To distill the impact of noisy labels , the related work either filters out suspiciously noisy data , derives robust loss functions or tries to proactively correct labels . Symmetric Cross entropy Loss ( SCL ) is shown effective in combating label noise especially for hard classes by combing the regular with the reverse cross entropy . The former avoids overfitting and the latter is resilient to label noise . Given its promising results , there is yet to have a clear principle on how to weight the regular and reverse cross entropy terms , e.g. , at different noise rates and patterns . In contrast , Distilling ( Li et al. , 2017 ) and Golden Loss Correction ( GLC ) ( Hendrycks et al. , 2018 ) advocate to use a small clean data to improve the estimated corruption matrix . Specifically , GLC trains the deep model on both a clean and noisy set , whose loss is corrected through the corruption matrix . While the clean set is evenly chosen from all classes , the corrupted labels may appear unevenly across classes depending on the noise pattern ( Xiao et al. , 2015 ) . As the corrected loss of GLC does not differentiate the difficulty of classes , it may not learn those hard classes effectively . We propose GSL constructing the golden symmetric loss that dynamically weights regular/reverse cross entropy and corrects the label prediction based on the estimated corruption matrix . Similar to GLC , GSL leverages clean data to estimate the corruption matrix which is used to correct labels and decide the weights of the golden symmetric loss . As such , GSL can effectively differentiate the difficulty level of classes by adjusting the weights and mitigate the impact of noise overfitting via the golden symmetric cross entropy . Specifically , we use the noise rate and noise diversity to adaptively tune the weights of modified cross entropy and reverse cross entropy . We prove that modified cross entropy by using confusion matrix is noise tolerant same as the reverse cross entropy . Motivation example . We demonstrate the advantages and disadvantages of GLC and SCL , and the their combination ( the proposed GSL ) through the example of learning convolution networks on CIFAR-10 injected with 60 % symmetric noise . The experimental setup is detailed in §6 . Figure 1 shows the corruption matrix of the injected noise and the confusion matrices from the predictions of SCL , GLC , and GSL . Even if the injected noise is symmetric across all classes ( see Figure 1a ) , prediction errors are distributed asymmetrically across the classes ( see Figure 1b ) . Though GLC can achieve a lower average error rate than SCL ( reflected in darker diagonal elements on average ) , it performs worse in hard classes , e.g. , class 4 ( cat ) and class 6 ( dog ) ( difference in blue shades across the diagonal elements ) . By setting up proper weights for two types of cross entropy , GSL is able to achieve both superior average and per class accuracy . 2 RELATED WORK . Enhancing the robustness of deep models against noisy labels is an active research area . The massive datasets needed to train deep models are commonly found corrupted , ( Wang et al. , 2018 ) , severely degrading the achievable accuracy , ( Zhang et al. , 2017 ) . The impact of label noise on deep neural networks is first characterized by the theoretical testing accuracy over a limited set of noise patterns ( Chen et al. , 2019 ) . ( Vahdat , 2017 ) suggest an undirected graph model for modeling label noise in deep neural networks and indicate symmetric noise to be more challenging than asymmetric . Solutions of the prior art can be categorized into three directions : ( i ) filtering out noisy labels : ( Malach & Shalev-Shwartz , 2017 ; Han et al. , 2018b ; Yu et al. , 2019 ; Wang et al. , 2018 ) ; ( ii ) correcting noisy labels : ( Patrini et al. , 2017 ; Hendrycks et al. , 2018 ; Li et al. , 2017 ) ; and ( iii ) deriving noise resilient loss functions : ( Ma et al. , 2018 ; Konstantinov & Lampert , 2019 ) . Noise Resilient Loss Function . The loss function is modified to enhance the robustness to label noise by introducing new loss functions , ( Ghosh et al. , 2017 ; Wang et al. , 2019 ) , or adjusting the weights of noisy data instances , ( Ren et al. , 2018b ; Konstantinov & Lampert , 2019 ; Ma et al. , 2018 ) . Mean Absolute Error ( MAE ) ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ) and General Cross Entropy loss ( Zhang & Sabuncu , 2018 ) are proposed as a noise resilient alternative but at the cost of slow convergence . To avoid overfitting to noise , D2L ( Ma et al. , 2018 ) uses the subspace dimensionality to assign weights to each data point , whereas Konstantinov ( Konstantinov & Lampert , 2019 ) determines the loss weights based on the trustworthiness level of data sources . ( Wang et al. , 2019 ) propose symmetric cross-entropy loss that combines a new term of reverse cross entropy with traditional cross entropy via constant weights on both terms . Meta-Weight-Net ( Shu et al. , 2019 ) re-weights samples during optimizing loss function in the training process by using a multi-layer perceptron to predict the weight of each sample . With the same perspective , ( Ren et al. , 2018a ) uses the similarity of samples to the clean instances in the validation set for re-weighting them in loss function . Label correction . To avoid the data reduction caused by filtering , label correction methods adjust the predicted/given labels by using only noisy labels ( Patrini et al. , 2017 ; Tanaka et al. , 2018 ) or jointly with a small fraction of trusted data ( Veit et al. , 2017 ; Han et al. , 2018a ; Li et al. , 2017 ; Hendrycks et al. , 2018 ) . Reed et al . train the classifier by the “ new ” labels combining the raw and predicted labels without access to label ground truth . ( Patrini et al. , 2017 ) estimate the noise confusion matrix by first training a classifier on the noisy labels and then using the softmax probabilities . ( Veit et al. , 2017 ) acquire human-verified labels to train a cleaning network for correcting noisy labels of multi-label classification problems . ( Han et al. , 2018a ) estimate the noise transition probability by incorporating human assistance . ( Li et al. , 2017 ) and ( Hendrycks et al. , 2018 ) leverage a small set of clean data to estimate noise corruption matrix from the clean and noisy sets , respectively . DivideMix ( Li et al. , 2020 ) is a semi-supervised method , including two networks and Gaussian Mixture Model for sample selection . The proposed GSL combines resilient loss function and label correction by curating a small fraction of trusted data . We solicit a subset of informative data instances to estimate the confusion matrix and provide a minimum supervision on noisy labels . We also provide a heuristic to adaptively tune the weights of golden symmetric loss according to the noise characteristics of the dataset . 3 GOLDEN SYMMETRIC LOSS . Consider the classification problem having dataset D̃ = { ( xn , ỹn ) } Nn=1 where xn ∈ X ⊂ Rd denotes the nth observed sample , and ỹn ∈ Y : = { 1 , ... , K } the corresponding given label over K classes . Hereon n is ignore for the simplicity . ỹ is affected by label noise . The label corruption process is characterised by a corruption matrix Cij = P ( ỹ = j|y = i ) for i = 1 , . . . , K and j = 1 , . . . , K where y is the true label . Synthetic noise patterns are expressed as a label corruption probability ε plus a noise label distribution . Let g ( · , θ ) denote a neural network-based classifier parameterized by θ . For each data point x , f ( · , θ ) predicts the probability for each class label k : p ( k|x ) = e zk∑K j=1 e zj where zj are the logits . Symmetric Cross Entropy . Let q ( k|x ) denote the ground truth probability distribution over the K class labels where q ( k|x ) = 1 for k equal to the true class y and q ( k|x ) = 0 for all k 6= y . The cross entropy loss ( ` ce ) and reverse cross entropy loss1 ( ` rce ) for sample x are : ` ce = − K∑ k=1 q ( k|x ) log p ( k|x ) , ` rce = − K∑ k=1 p ( k|x ) log q ( k|x ) . ( 1 ) ( Wang et al. , 2019 ) combine cross entropy and reverse cross entropy into the symmetric cross entropy : lsl = α ` ce + β ` rce . ( 2 ) where α and β are hyperparameters . On the one hand cross entropy loss is not robust to noise ( Ghosh et al. , 2017 ) but achieves good convergence ( Zhang & Sabuncu , 2018 ) . On the other hand reverse cross entropy is tolerant to noise ( Wang et al. , 2019 ) . Estimating Noise Corruption Matrix . We estimate the noise corruption matrix as in ( Hendrycks et al. , 2018 ) . The method fosters training a first classifier g ( · , Θ ) on noisy data to approximate the elements Cij of the noise corruption matrix via a small fraction of trusted data D with known true label y . Practically given Ai the subset of trusted data with label of class i { Ai ⊂ D : y = i } , the elements of C can be approximated by : Ĉij = P ( ỹ = j|y = i ) ≈ 1 |Ai| ∑ x∈Ai g ( ỹ = j|x , Θ ) ( 3 ) where g ( ỹ = j|x , Θ ) denotes predicted probability of x having class label j . That is Ĉij is computed as the mean predicted probability of class j for all trusted data points having true label of class i . Training with Corrected Labels . Let Ĉ be the estimated noise corruption matrix . Using the method in ( Patrini et al. , 2017 ) , we increase the noise resilience by correcting the predictions of the 1To avoid problems with the logarithm , zero values of q are replaced by a small positive value , i.e . 10−4 . classifier using Ĉ . Let p̂ be the corrected predicted probabilities p̂ = ĈT p , i.e . for data point x : p̂ ( k|x ) = ∑K i=1 Ĉikp ( i|x ) for k = 1 , . . . , K. We enhance the regular cross entropy term . Applying the prediction correction to both terms holds lower benefits . We evaluate this empirically with extensive experiments on datasets of text , i.e . Twitter in Figure . 2a , and images , i.e . CIFAR10 and CIFAR100 in Appendix A.Experiment details can be found in §6 . We consider different datasets , noise rates , noise types and fractions of trusted data . We see that in all cases , except one with a difference < 0.3 % , correcting only the cross entropy ( ce-only ) holds better results than correcting only the reverse cross entropy ( rce-only ) or correcting both . Focusing on Figure . 2a , ce-only improves accuracy by up to 5 % and 8 % for bimodal and symmetric noise , respectively . In case of CIFAR-10 and CIFAR-100 datasets the improvements are more pronounced with up to 11 % and 50 % respectively . Golden Symmetric Loss . Towards a more effective and robust learning we propose to leverage the estimated noise corruption matrix Ĉ to tune the two loss terms based on the observed noise pattern . α and β can significantly impact the final model accuracy . Tuning these parameters is no mean feat as different datasets affected by different noise patterns benefit from different optimal values ( Wang et al. , 2019 ) . Again we show this behavior by training a 2-layer FC neural network on the Twitter dataset under eleven different ( α , β ) combinations and two noise patterns with 80 % noise . Figure . 2b reports for each noise pattern the evolution over the training epochs of the test accuracy for the best and worst ( α , β ) -pair . For bimodal noise even with a small number of trials , the impact of ( α , β ) ranges from an accuracy close to 60 % all the way down to almost 0 % . Moreover only few ( two out of eleven ) ( α , β ) -pairs hold accuracy close to 60 % . For symmetric noise the tuning impact is lower ( limited between 70 % and 80 % ) but the best and worst ( α , β ) -pair differ from the bimodal noise case . This underlines both the importance and difficulty of tuning ( α , β ) . Motivated by the high impact of α and β , we propose to dynamically weight the regular and reverse cross entropy terms . Let A ( ) and B ( ) be weighting functions mapping Ĉ −→ R we define a new loss function : ` GSL = A ( Ĉ ) ` ce +B ( Ĉ ) ` rce ( 4 ) We call this new loss function golden symmetric loss . A ( ) and B ( ) should capture not only the intensity of the noise pattern , but also the diversity of the noise pattern ( see Figure . 2b ) . Determining Weights of Golden Symmetric Loss ( A ( ) and B ( ) ) . In general the more intense and asymmetric the noise pattern , the lower the weight values should be . Since the final loss function learns from both dirty and clean data ( see next paragraph ) , lower values of α and β reduce the influence of dirty data over the one of clean data . Hence , we design A ( ) and B ( ) to capture both noise intensity and diversity . The intensity is given by the noise rate ε ∈ [ 0 , . . . , 1 ] , i.e . one minus the average of the diagonal elements of Ĉ . The diversity is measured via Jain ’ s fairness index J ( x1 , x2 , . . . , xn ) , ( ∑n i=n xi ) 2 /n ∑n i=n x 2 i . We choose J because it bounds the diversity on a similar scale as ε between 1 ( all equal , full symmetry ) down to 1/n ( highest asymmetry ) . We apply J on all the K ( K − 1 ) noise , i.e . off the diagonal , elements of Ĉ : J = ( ∑K i=1 ∑K j=1 , j 6=i Ĉij ) 2 K ( K − 1 ) ∑K i=1 ∑K j=1 , j 6=i Ĉ 2 ij ( 5 ) Under review as a conference paper at ICLR 2021 < latexit sha1_base64= '' Vx5fjB2iNIUIe3dyoWc60cmkxhY= '' > AAAB/HicbVDLSsNAFL2pr1pf0S7dDBbBVUmkoO6KunBZwT6gCWUymbZDJw9mJkII8VfcuFDErR/izr9x0mahrQcGDufcyz1zvJgzqSzr26isrW9sblW3azu7e/sH5uFRT0aJILRLIh6JgYcl5SykXcUUp4NYUBx4nPa92U3h9x+pkCwKH1QaUzfAk5CNGcFKSyOz7ijGfZo5AVZTgnl2m+cjs2E1rTnQKrFL0oASnZH55fgRSQIaKsKxlEPbipWbYaEY4TSvOYmkMSYzPKFDTUMcUOlm8/A5OtWKj8aR0C9UaK7+3shwIGUaeHqyyCiXvUL8zxsmanzpZiyME0VDsjg0TjhSESqaQD4TlCieaoKJYDorIlMsMFG6r5ouwV7+8irpnTftVvPqvtVoX5d1VOEYTuAMbLiANtxBB7pAIIVneIU348l4Md6Nj8VoxSh36vAHxucPb8aVTQ== < /latexit > < latexit sha1_base64= '' 5EK7A9JpdERxBEm8oBLZpmfNc5E= '' > AAAB8nicbVDLSgMxFL1TX7W+qi7dBIvgqsxIQd0VdeGygn3AdCiZNNOGZpIhyQhl6Ge4caGIW7/GnX9jpp2Fth4IHM65l5x7woQzbVz32ymtrW9sbpW3Kzu7e/sH1cOjjpapIrRNJJeqF2JNORO0bZjhtJcoiuOQ0244uc397hNVmknxaKYJDWI8EixiBBsr+f0YmzHBPLubDao1t+7OgVaJV5AaFGgNql/9oSRpTIUhHGvte25iggwrwwins0o/1TTBZIJH1LdU4JjqIJtHnqEzqwxRJJV9wqC5+nsjw7HW0zi0k3lEvezl4n+en5roKsiYSFJDBVl8FKUcGYny+9GQKUoMn1qCiWI2KyJjrDAxtqWKLcFbPnmVdC7qXqN+/dCoNW+KOspwAqdwDh5cQhPuoQVtICDhGV7hzTHOi/PufCxGS06xcwx/4Hz+AHi9kWY= < /latexit > < latexit sha1_base64= '' 6Z46ps9B4Y0rmQSBC2/4tqqB8jw= '' > AAAB8nicbVDLSgMxFL1TX7W+qi7dBIvgqsyUgror6MJlBfuA6VAyaaYNzSRDkhHK0M9w40IRt36NO//GTDsLbT0QOJxzLzn3hAln2rjut1Pa2Nza3invVvb2Dw6PqscnXS1TRWiHSC5VP8SaciZoxzDDaT9RFMchp71wepv7vSeqNJPi0cwSGsR4LFjECDZW8gcxNhOCeXY3H1Zrbt1dAK0TryA1KNAeVr8GI0nSmApDONba99zEBBlWhhFO55VBqmmCyRSPqW+pwDHVQbaIPEcXVhmhSCr7hEEL9fdGhmOtZ3FoJ/OIetXLxf88PzXRdZAxkaSGCrL8KEo5MhLl96MRU5QYPrMEE8VsVkQmWGFibEsVW4K3evI66TbqXrN+89CstRpFHWU4g3O4BA+uoAX30IYOEJDwDK/w5hjnxXl3PpajJafYOYU/cD5/AHPtkVY= < /latexit > < latexit sha1_base64= '' Vx5fjB2iNIUIe3dyoWc60cmkxhY= '' > AAAB/HicbVDLSsNAFL2pr1pf0S7dDBbBVUmkoO6KunBZwT6gCWUymbZDJw9mJkII8VfcuFDErR/izr9x0mahrQcGDufcyz1zvJgzqSzr26isrW9sblW3azu7e/sH5uFRT0aJILRLIh6JgYcl5SykXcUUp4NYUBx4nPa92U3h9x+pkCwKH1QaUzfAk5CNGcFKSyOz7ijGfZo5AVZTgnl2m+cjs2E1rTnQKrFL0oASnZH55fgRSQIaKsKxlEPbipWbYaEY4TSvOYmkMSYzPKFDTUMcUOlm8/A5OtWKj8aR0C9UaK7+3shwIGUaeHqyyCiXvUL8zxsmanzpZiyME0VDsjg0TjhSESqaQD4TlCieaoKJYDorIlMsMFG6r5ouwV7+8irpnTftVvPqvtVoX5d1VOEYTuAMbLiANtxBB7pAIIVneIU348l4Md6Nj8VoxSh36vAHxucPb8aVTQ== < /latexit > < latexit sha1_base64= '' BA90eKkxRIs8+kXspOPTEZuRZ2I= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lKQb0VevFYwX5AG8pmu2mXbjZhdyKU0B/hxYMiXv093vw3btsctPXBwOO9GWbmBYkUBl332ylsbe/s7hX3SweHR8cn5dOzjolTzXibxTLWvYAaLoXibRQoeS/RnEaB5N1g2lz43SeujYjVI84S7kd0rEQoGEUrdQcTillzPixX3Kq7BNkkXk4qkKM1LH8NRjFLI66QSWpM33MT9DOqUTDJ56VBanhC2ZSOed9SRSNu/Gx57pxcWWVEwljbUkiW6u+JjEbGzKLAdkYUJ2bdW4j/ef0Uw1s/EypJkSu2WhSmkmBMFr+TkdCcoZxZQpkW9lbCJlRThjahkg3BW395k3RqVa9evXuoVxq1PI4iXMAlXIMHN9CAe2hBGxhM4Rle4c1JnBfn3flYtRacfOYc/sD5/AFew4+Q < /latexit > < latexit sha1_base64= '' nmAgKmMr6cYo77NyfdA16hufLK4= '' > AAACDHicbVDJSgNBFOyJW4xb1KOXxiAklzAjARURgl48KpgoZIbQ03lJmvQsdL+RDGM+wIu/4sWDIl79AG/+jZ3l4FbQUFTV4/UrP5ZCo21/Wrm5+YXFpfxyYWV1bX2juLnV1FGiODR4JCN14zMNUoTQQIESbmIFLPAlXPuDs7F/fQtKiyi8wjQGL2C9UHQFZ2ikdrHUKw8r9IS6fYZZPCq7KGQHsnR0Nzym7lUfkFVMyq7aE9C/xJmREpnhol38cDsRTwIIkUumdcuxY/QyplBwCaOCm2iIGR+wHrQMDVkA2ssmx4zonlE6tBsp80KkE/X7RMYCrdPAN8mAYV//9sbif14rwe6hl4kwThBCPl3UTSTFiI6boR2hgKNMDWFcCfNXyvtMMY6mv4Ipwfl98l/S3K86terRZa1UP53VkSc7ZJeUiUMOSJ2ckwvSIJzck0fyTF6sB+vJerXeptGcNZvZJj9gvX8BV6WalA== < /latexit > < latexit sha1_base64= '' xSckbF/QAaF6UST6bVBnYN8bV/w= '' > AAACBXicbVDLSsNAFJ3UV62vqEtdDBahBSmJFNRdoQtduKjYFzSxTKaTdujkwcxEDCEbN/6KGxeKuPUf3Pk3TtsstPXAhcM593LvPU7IqJCG8a3llpZXVtfy64WNza3tHX13ry2CiGPSwgELeNdBgjDqk5akkpFuyAnyHEY6zrg+8Tv3hAsa+E0Zh8T20NCnLsVIKqmvH1qEsX5yeXudlqwRkkk9vWu6pYfyCYzLfb1oVIwp4CIxM1IEGRp9/csaBDjyiC8xQ0L0TCOUdoK4pJiRtGBFgoQIj9GQ9BT1kUeEnUy/SOGxUgbQDbgqX8Kp+nsiQZ4QseeoTg/JkZj3JuJ/Xi+S7rmdUD+MJPHxbJEbMSgDOIkEDignWLJYEYQ5VbdCPEIcYamCK6gQzPmXF0n7tGJWKxc31WKtmsWRBwfgCJSACc5ADVyBBmgBDB7BM3gFb9qT9qK9ax+z1pyWzeyDP9A+fwC0ypdi < /latexit > < latexit sha1_base64= '' sQ7SxJkZVC7sJv3AogpxYECSHiY= '' > AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCC1ISKai3ghePFWwttCFstpN26eaD3Y00hPwVLx4U8eof8ea/cdvmoK0PBh7vzTAzz4s5k8qyvo3SxubW9k55t7K3f3B4ZB5XezJKBIUujXgk+h6RwFkIXcUUh34sgAQeh0dvejv3H59ASBaFDyqNwQnIOGQ+o0RpyTWrQ+DczSjkdb8+a1zgtOGaNatpLYDXiV2QGirQcc2v4SiiSQChopxIObCtWDkZEYpRDnllmEiICZ2SMQw0DUkA0skWt+f4XCsj7EdCV6jwQv09kZFAyjTwdGdA1ESuenPxP2+QKP/ayVgYJwpCulzkJxyrCM+DwCMmgCqeakKoYPpWTCdEEKp0XBUdgr368jrpXTbtVvPmvlVrt4o4yugUnaE6stEVaqM71EFdRNEMPaNX9GbkxovxbnwsW0tGMXOC/sD4/AFpe5Na < /latexit > For symmetric noise J = 1 , the more asymmetric the smaller J . Final weights proportional to J , ε . Putting It All Together . As a final step , to maximize the utility of the trusted data , we foster D as additional trusted training data for f ( ) . Since D contains the true labels y no prediction correction is applied . Hence , the overall loss function for data points from both D and D̃ is : l = { −A ( Ĉ ) ∑K k=1 q ( k|x ) log ( ∑K i=1 Ĉikp ( i|x ) ) −B ( Ĉ ) ∑K k=1 p ( k|x ) log q ( k|x ) , x ∈ D̃ − ∑K k=1 q ( k|x ) log p ( k|x ) , x ∈ D. ( 6 ) Figure . 3 summarises visually the training process divided into two main steps : ( i ) estimating noise corruption matrix through the first network g trained on untrusted dataset D̃ and ( ii ) training classifier f on both untrusted D̃ and trusted D through the golden symmetric loss .

Summary: The paper proposes a golden symmetric loss method that combines cross entropy loss and reverse cross entropy loss, and at the same time performs forward loss correction for cross entropy loss, under the problem where we have noisy training labels and clean training labels. It shows theoretical results showing the robustness of cross entropy loss under forward loss correction, and shows that it can be regarded as a generalization of Ghosh et al. 2017's analysis. Empirically, it shows how loss correction on reversed cross entropy or the symmetric cross entropy performs worse compared with only cross entropy, and propose to use forward loss correction only on cross entropy term.