Robust Learning via Golden Symmetric Loss of (un)Trusted Labels

1 INTRODUCTION . Diverse datasets collected from the public domain which power up deep learning models present new challenges – highly noisy labels . It is not only time consuming to collect labels but also difficult to ensure a consistent label quality due to various annotation errors ( Patrini et al. , 2017 ) and adversarial attacks ( Goodfellow et al. , 2015 ) . The large capacity of deep learning models enables effective learning from complex datasets but also suffers from overfitting to the noise structure in the dataset . The curse of memorization effect ( Jiang et al. , 2018 ) can degrade the accuracy of deep learning models in the presence of highly noisy labels . For example , in ( Zhang et al. , 2017 ) the accuracy of AlexNet to classify CIFAR10 images drops from 77 % to 10 % , when there are randomly flipped labels . Designing learning models that can robustly train on noisy labels is thus imperative . To distill the impact of noisy labels , the related work either filters out suspiciously noisy data , derives robust loss functions or tries to proactively correct labels . Symmetric Cross entropy Loss ( SCL ) is shown effective in combating label noise especially for hard classes by combing the regular with the reverse cross entropy . The former avoids overfitting and the latter is resilient to label noise . Given its promising results , there is yet to have a clear principle on how to weight the regular and reverse cross entropy terms , e.g. , at different noise rates and patterns . In contrast , Distilling ( Li et al. , 2017 ) and Golden Loss Correction ( GLC ) ( Hendrycks et al. , 2018 ) advocate to use a small clean data to improve the estimated corruption matrix . Specifically , GLC trains the deep model on both a clean and noisy set , whose loss is corrected through the corruption matrix . While the clean set is evenly chosen from all classes , the corrupted labels may appear unevenly across classes depending on the noise pattern ( Xiao et al. , 2015 ) . As the corrected loss of GLC does not differentiate the difficulty of classes , it may not learn those hard classes effectively . We propose GSL constructing the golden symmetric loss that dynamically weights regular/reverse cross entropy and corrects the label prediction based on the estimated corruption matrix . Similar to GLC , GSL leverages clean data to estimate the corruption matrix which is used to correct labels and decide the weights of the golden symmetric loss . As such , GSL can effectively differentiate the difficulty level of classes by adjusting the weights and mitigate the impact of noise overfitting via the golden symmetric cross entropy . Specifically , we use the noise rate and noise diversity to adaptively tune the weights of modified cross entropy and reverse cross entropy . We prove that modified cross entropy by using confusion matrix is noise tolerant same as the reverse cross entropy . Motivation example . We demonstrate the advantages and disadvantages of GLC and SCL , and the their combination ( the proposed GSL ) through the example of learning convolution networks on CIFAR-10 injected with 60 % symmetric noise . The experimental setup is detailed in §6 . Figure 1 shows the corruption matrix of the injected noise and the confusion matrices from the predictions of SCL , GLC , and GSL . Even if the injected noise is symmetric across all classes ( see Figure 1a ) , prediction errors are distributed asymmetrically across the classes ( see Figure 1b ) . Though GLC can achieve a lower average error rate than SCL ( reflected in darker diagonal elements on average ) , it performs worse in hard classes , e.g. , class 4 ( cat ) and class 6 ( dog ) ( difference in blue shades across the diagonal elements ) . By setting up proper weights for two types of cross entropy , GSL is able to achieve both superior average and per class accuracy . 2 RELATED WORK . Enhancing the robustness of deep models against noisy labels is an active research area . The massive datasets needed to train deep models are commonly found corrupted , ( Wang et al. , 2018 ) , severely degrading the achievable accuracy , ( Zhang et al. , 2017 ) . The impact of label noise on deep neural networks is first characterized by the theoretical testing accuracy over a limited set of noise patterns ( Chen et al. , 2019 ) . ( Vahdat , 2017 ) suggest an undirected graph model for modeling label noise in deep neural networks and indicate symmetric noise to be more challenging than asymmetric . Solutions of the prior art can be categorized into three directions : ( i ) filtering out noisy labels : ( Malach & Shalev-Shwartz , 2017 ; Han et al. , 2018b ; Yu et al. , 2019 ; Wang et al. , 2018 ) ; ( ii ) correcting noisy labels : ( Patrini et al. , 2017 ; Hendrycks et al. , 2018 ; Li et al. , 2017 ) ; and ( iii ) deriving noise resilient loss functions : ( Ma et al. , 2018 ; Konstantinov & Lampert , 2019 ) . Noise Resilient Loss Function . The loss function is modified to enhance the robustness to label noise by introducing new loss functions , ( Ghosh et al. , 2017 ; Wang et al. , 2019 ) , or adjusting the weights of noisy data instances , ( Ren et al. , 2018b ; Konstantinov & Lampert , 2019 ; Ma et al. , 2018 ) . Mean Absolute Error ( MAE ) ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ) and General Cross Entropy loss ( Zhang & Sabuncu , 2018 ) are proposed as a noise resilient alternative but at the cost of slow convergence . To avoid overfitting to noise , D2L ( Ma et al. , 2018 ) uses the subspace dimensionality to assign weights to each data point , whereas Konstantinov ( Konstantinov & Lampert , 2019 ) determines the loss weights based on the trustworthiness level of data sources . ( Wang et al. , 2019 ) propose symmetric cross-entropy loss that combines a new term of reverse cross entropy with traditional cross entropy via constant weights on both terms . Meta-Weight-Net ( Shu et al. , 2019 ) re-weights samples during optimizing loss function in the training process by using a multi-layer perceptron to predict the weight of each sample . With the same perspective , ( Ren et al. , 2018a ) uses the similarity of samples to the clean instances in the validation set for re-weighting them in loss function . Label correction . To avoid the data reduction caused by filtering , label correction methods adjust the predicted/given labels by using only noisy labels ( Patrini et al. , 2017 ; Tanaka et al. , 2018 ) or jointly with a small fraction of trusted data ( Veit et al. , 2017 ; Han et al. , 2018a ; Li et al. , 2017 ; Hendrycks et al. , 2018 ) . Reed et al . train the classifier by the “ new ” labels combining the raw and predicted labels without access to label ground truth . ( Patrini et al. , 2017 ) estimate the noise confusion matrix by first training a classifier on the noisy labels and then using the softmax probabilities . ( Veit et al. , 2017 ) acquire human-verified labels to train a cleaning network for correcting noisy labels of multi-label classification problems . ( Han et al. , 2018a ) estimate the noise transition probability by incorporating human assistance . ( Li et al. , 2017 ) and ( Hendrycks et al. , 2018 ) leverage a small set of clean data to estimate noise corruption matrix from the clean and noisy sets , respectively . DivideMix ( Li et al. , 2020 ) is a semi-supervised method , including two networks and Gaussian Mixture Model for sample selection . The proposed GSL combines resilient loss function and label correction by curating a small fraction of trusted data . We solicit a subset of informative data instances to estimate the confusion matrix and provide a minimum supervision on noisy labels . We also provide a heuristic to adaptively tune the weights of golden symmetric loss according to the noise characteristics of the dataset . 3 GOLDEN SYMMETRIC LOSS . Consider the classification problem having dataset D̃ = { ( xn , ỹn ) } Nn=1 where xn ∈ X ⊂ Rd denotes the nth observed sample , and ỹn ∈ Y : = { 1 , ... , K } the corresponding given label over K classes . Hereon n is ignore for the simplicity . ỹ is affected by label noise . The label corruption process is characterised by a corruption matrix Cij = P ( ỹ = j|y = i ) for i = 1 , . . . , K and j = 1 , . . . , K where y is the true label . Synthetic noise patterns are expressed as a label corruption probability ε plus a noise label distribution . Let g ( · , θ ) denote a neural network-based classifier parameterized by θ . For each data point x , f ( · , θ ) predicts the probability for each class label k : p ( k|x ) = e zk∑K j=1 e zj where zj are the logits . Symmetric Cross Entropy . Let q ( k|x ) denote the ground truth probability distribution over the K class labels where q ( k|x ) = 1 for k equal to the true class y and q ( k|x ) = 0 for all k 6= y . The cross entropy loss ( ` ce ) and reverse cross entropy loss1 ( ` rce ) for sample x are : ` ce = − K∑ k=1 q ( k|x ) log p ( k|x ) , ` rce = − K∑ k=1 p ( k|x ) log q ( k|x ) . ( 1 ) ( Wang et al. , 2019 ) combine cross entropy and reverse cross entropy into the symmetric cross entropy : lsl = α ` ce + β ` rce . ( 2 ) where α and β are hyperparameters . On the one hand cross entropy loss is not robust to noise ( Ghosh et al. , 2017 ) but achieves good convergence ( Zhang & Sabuncu , 2018 ) . On the other hand reverse cross entropy is tolerant to noise ( Wang et al. , 2019 ) . Estimating Noise Corruption Matrix . We estimate the noise corruption matrix as in ( Hendrycks et al. , 2018 ) . The method fosters training a first classifier g ( · , Θ ) on noisy data to approximate the elements Cij of the noise corruption matrix via a small fraction of trusted data D with known true label y . Practically given Ai the subset of trusted data with label of class i { Ai ⊂ D : y = i } , the elements of C can be approximated by : Ĉij = P ( ỹ = j|y = i ) ≈ 1 |Ai| ∑ x∈Ai g ( ỹ = j|x , Θ ) ( 3 ) where g ( ỹ = j|x , Θ ) denotes predicted probability of x having class label j . That is Ĉij is computed as the mean predicted probability of class j for all trusted data points having true label of class i . Training with Corrected Labels . Let Ĉ be the estimated noise corruption matrix . Using the method in ( Patrini et al. , 2017 ) , we increase the noise resilience by correcting the predictions of the 1To avoid problems with the logarithm , zero values of q are replaced by a small positive value , i.e . 10−4 . classifier using Ĉ . Let p̂ be the corrected predicted probabilities p̂ = ĈT p , i.e . for data point x : p̂ ( k|x ) = ∑K i=1 Ĉikp ( i|x ) for k = 1 , . . . , K. We enhance the regular cross entropy term . Applying the prediction correction to both terms holds lower benefits . We evaluate this empirically with extensive experiments on datasets of text , i.e . Twitter in Figure . 2a , and images , i.e . CIFAR10 and CIFAR100 in Appendix A.Experiment details can be found in §6 . We consider different datasets , noise rates , noise types and fractions of trusted data . We see that in all cases , except one with a difference < 0.3 % , correcting only the cross entropy ( ce-only ) holds better results than correcting only the reverse cross entropy ( rce-only ) or correcting both . Focusing on Figure . 2a , ce-only improves accuracy by up to 5 % and 8 % for bimodal and symmetric noise , respectively . In case of CIFAR-10 and CIFAR-100 datasets the improvements are more pronounced with up to 11 % and 50 % respectively . Golden Symmetric Loss . Towards a more effective and robust learning we propose to leverage the estimated noise corruption matrix Ĉ to tune the two loss terms based on the observed noise pattern . α and β can significantly impact the final model accuracy . Tuning these parameters is no mean feat as different datasets affected by different noise patterns benefit from different optimal values ( Wang et al. , 2019 ) . Again we show this behavior by training a 2-layer FC neural network on the Twitter dataset under eleven different ( α , β ) combinations and two noise patterns with 80 % noise . Figure . 2b reports for each noise pattern the evolution over the training epochs of the test accuracy for the best and worst ( α , β ) -pair . For bimodal noise even with a small number of trials , the impact of ( α , β ) ranges from an accuracy close to 60 % all the way down to almost 0 % . Moreover only few ( two out of eleven ) ( α , β ) -pairs hold accuracy close to 60 % . For symmetric noise the tuning impact is lower ( limited between 70 % and 80 % ) but the best and worst ( α , β ) -pair differ from the bimodal noise case . This underlines both the importance and difficulty of tuning ( α , β ) . Motivated by the high impact of α and β , we propose to dynamically weight the regular and reverse cross entropy terms . Let A ( ) and B ( ) be weighting functions mapping Ĉ −→ R we define a new loss function : ` GSL = A ( Ĉ ) ` ce +B ( Ĉ ) ` rce ( 4 ) We call this new loss function golden symmetric loss . A ( ) and B ( ) should capture not only the intensity of the noise pattern , but also the diversity of the noise pattern ( see Figure . 2b ) . Determining Weights of Golden Symmetric Loss ( A ( ) and B ( ) ) . In general the more intense and asymmetric the noise pattern , the lower the weight values should be . Since the final loss function learns from both dirty and clean data ( see next paragraph ) , lower values of α and β reduce the influence of dirty data over the one of clean data . Hence , we design A ( ) and B ( ) to capture both noise intensity and diversity . The intensity is given by the noise rate ε ∈ [ 0 , . . . , 1 ] , i.e . one minus the average of the diagonal elements of Ĉ . The diversity is measured via Jain ’ s fairness index J ( x1 , x2 , . . . , xn ) , ( ∑n i=n xi ) 2 /n ∑n i=n x 2 i . We choose J because it bounds the diversity on a similar scale as ε between 1 ( all equal , full symmetry ) down to 1/n ( highest asymmetry ) . We apply J on all the K ( K − 1 ) noise , i.e . off the diagonal , elements of Ĉ : J = ( ∑K i=1 ∑K j=1 , j 6=i Ĉij ) 2 K ( K − 1 ) ∑K i=1 ∑K j=1 , j 6=i Ĉ 2 ij ( 5 ) Under review as a conference paper at ICLR 2021 < latexit sha1_base64= '' Vx5fjB2iNIUIe3dyoWc60cmkxhY= '' > AAAB/HicbVDLSsNAFL2pr1pf0S7dDBbBVUmkoO6KunBZwT6gCWUymbZDJw9mJkII8VfcuFDErR/izr9x0mahrQcGDufcyz1zvJgzqSzr26isrW9sblW3azu7e/sH5uFRT0aJILRLIh6JgYcl5SykXcUUp4NYUBx4nPa92U3h9x+pkCwKH1QaUzfAk5CNGcFKSyOz7ijGfZo5AVZTgnl2m+cjs2E1rTnQKrFL0oASnZH55fgRSQIaKsKxlEPbipWbYaEY4TSvOYmkMSYzPKFDTUMcUOlm8/A5OtWKj8aR0C9UaK7+3shwIGUaeHqyyCiXvUL8zxsmanzpZiyME0VDsjg0TjhSESqaQD4TlCieaoKJYDorIlMsMFG6r5ouwV7+8irpnTftVvPqvtVoX5d1VOEYTuAMbLiANtxBB7pAIIVneIU348l4Md6Nj8VoxSh36vAHxucPb8aVTQ== < /latexit > < latexit sha1_base64= '' 5EK7A9JpdERxBEm8oBLZpmfNc5E= '' > AAAB8nicbVDLSgMxFL1TX7W+qi7dBIvgqsxIQd0VdeGygn3AdCiZNNOGZpIhyQhl6Ge4caGIW7/GnX9jpp2Fth4IHM65l5x7woQzbVz32ymtrW9sbpW3Kzu7e/sH1cOjjpapIrRNJJeqF2JNORO0bZjhtJcoiuOQ0244uc397hNVmknxaKYJDWI8EixiBBsr+f0YmzHBPLubDao1t+7OgVaJV5AaFGgNql/9oSRpTIUhHGvte25iggwrwwins0o/1TTBZIJH1LdU4JjqIJtHnqEzqwxRJJV9wqC5+nsjw7HW0zi0k3lEvezl4n+en5roKsiYSFJDBVl8FKUcGYny+9GQKUoMn1qCiWI2KyJjrDAxtqWKLcFbPnmVdC7qXqN+/dCoNW+KOspwAqdwDh5cQhPuoQVtICDhGV7hzTHOi/PufCxGS06xcwx/4Hz+AHi9kWY= < /latexit > < latexit sha1_base64= '' 6Z46ps9B4Y0rmQSBC2/4tqqB8jw= '' > AAAB8nicbVDLSgMxFL1TX7W+qi7dBIvgqsyUgror6MJlBfuA6VAyaaYNzSRDkhHK0M9w40IRt36NO//GTDsLbT0QOJxzLzn3hAln2rjut1Pa2Nza3invVvb2Dw6PqscnXS1TRWiHSC5VP8SaciZoxzDDaT9RFMchp71wepv7vSeqNJPi0cwSGsR4LFjECDZW8gcxNhOCeXY3H1Zrbt1dAK0TryA1KNAeVr8GI0nSmApDONba99zEBBlWhhFO55VBqmmCyRSPqW+pwDHVQbaIPEcXVhmhSCr7hEEL9fdGhmOtZ3FoJ/OIetXLxf88PzXRdZAxkaSGCrL8KEo5MhLl96MRU5QYPrMEE8VsVkQmWGFibEsVW4K3evI66TbqXrN+89CstRpFHWU4g3O4BA+uoAX30IYOEJDwDK/w5hjnxXl3PpajJafYOYU/cD5/AHPtkVY= < /latexit > < latexit sha1_base64= '' Vx5fjB2iNIUIe3dyoWc60cmkxhY= '' > AAAB/HicbVDLSsNAFL2pr1pf0S7dDBbBVUmkoO6KunBZwT6gCWUymbZDJw9mJkII8VfcuFDErR/izr9x0mahrQcGDufcyz1zvJgzqSzr26isrW9sblW3azu7e/sH5uFRT0aJILRLIh6JgYcl5SykXcUUp4NYUBx4nPa92U3h9x+pkCwKH1QaUzfAk5CNGcFKSyOz7ijGfZo5AVZTgnl2m+cjs2E1rTnQKrFL0oASnZH55fgRSQIaKsKxlEPbipWbYaEY4TSvOYmkMSYzPKFDTUMcUOlm8/A5OtWKj8aR0C9UaK7+3shwIGUaeHqyyCiXvUL8zxsmanzpZiyME0VDsjg0TjhSESqaQD4TlCieaoKJYDorIlMsMFG6r5ouwV7+8irpnTftVvPqvtVoX5d1VOEYTuAMbLiANtxBB7pAIIVneIU348l4Md6Nj8VoxSh36vAHxucPb8aVTQ== < /latexit > < latexit sha1_base64= '' BA90eKkxRIs8+kXspOPTEZuRZ2I= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lKQb0VevFYwX5AG8pmu2mXbjZhdyKU0B/hxYMiXv093vw3btsctPXBwOO9GWbmBYkUBl332ylsbe/s7hX3SweHR8cn5dOzjolTzXibxTLWvYAaLoXibRQoeS/RnEaB5N1g2lz43SeujYjVI84S7kd0rEQoGEUrdQcTillzPixX3Kq7BNkkXk4qkKM1LH8NRjFLI66QSWpM33MT9DOqUTDJ56VBanhC2ZSOed9SRSNu/Gx57pxcWWVEwljbUkiW6u+JjEbGzKLAdkYUJ2bdW4j/ef0Uw1s/EypJkSu2WhSmkmBMFr+TkdCcoZxZQpkW9lbCJlRThjahkg3BW395k3RqVa9evXuoVxq1PI4iXMAlXIMHN9CAe2hBGxhM4Rle4c1JnBfn3flYtRacfOYc/sD5/AFew4+Q < /latexit > < latexit sha1_base64= '' nmAgKmMr6cYo77NyfdA16hufLK4= '' > AAACDHicbVDJSgNBFOyJW4xb1KOXxiAklzAjARURgl48KpgoZIbQ03lJmvQsdL+RDGM+wIu/4sWDIl79AG/+jZ3l4FbQUFTV4/UrP5ZCo21/Wrm5+YXFpfxyYWV1bX2juLnV1FGiODR4JCN14zMNUoTQQIESbmIFLPAlXPuDs7F/fQtKiyi8wjQGL2C9UHQFZ2ikdrHUKw8r9IS6fYZZPCq7KGQHsnR0Nzym7lUfkFVMyq7aE9C/xJmREpnhol38cDsRTwIIkUumdcuxY/QyplBwCaOCm2iIGR+wHrQMDVkA2ssmx4zonlE6tBsp80KkE/X7RMYCrdPAN8mAYV//9sbif14rwe6hl4kwThBCPl3UTSTFiI6boR2hgKNMDWFcCfNXyvtMMY6mv4Ipwfl98l/S3K86terRZa1UP53VkSc7ZJeUiUMOSJ2ckwvSIJzck0fyTF6sB+vJerXeptGcNZvZJj9gvX8BV6WalA== < /latexit > < latexit sha1_base64= '' xSckbF/QAaF6UST6bVBnYN8bV/w= '' > AAACBXicbVDLSsNAFJ3UV62vqEtdDBahBSmJFNRdoQtduKjYFzSxTKaTdujkwcxEDCEbN/6KGxeKuPUf3Pk3TtsstPXAhcM593LvPU7IqJCG8a3llpZXVtfy64WNza3tHX13ry2CiGPSwgELeNdBgjDqk5akkpFuyAnyHEY6zrg+8Tv3hAsa+E0Zh8T20NCnLsVIKqmvH1qEsX5yeXudlqwRkkk9vWu6pYfyCYzLfb1oVIwp4CIxM1IEGRp9/csaBDjyiC8xQ0L0TCOUdoK4pJiRtGBFgoQIj9GQ9BT1kUeEnUy/SOGxUgbQDbgqX8Kp+nsiQZ4QseeoTg/JkZj3JuJ/Xi+S7rmdUD+MJPHxbJEbMSgDOIkEDignWLJYEYQ5VbdCPEIcYamCK6gQzPmXF0n7tGJWKxc31WKtmsWRBwfgCJSACc5ADVyBBmgBDB7BM3gFb9qT9qK9ax+z1pyWzeyDP9A+fwC0ypdi < /latexit > < latexit sha1_base64= '' sQ7SxJkZVC7sJv3AogpxYECSHiY= '' > AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCC1ISKai3ghePFWwttCFstpN26eaD3Y00hPwVLx4U8eof8ea/cdvmoK0PBh7vzTAzz4s5k8qyvo3SxubW9k55t7K3f3B4ZB5XezJKBIUujXgk+h6RwFkIXcUUh34sgAQeh0dvejv3H59ASBaFDyqNwQnIOGQ+o0RpyTWrQ+DczSjkdb8+a1zgtOGaNatpLYDXiV2QGirQcc2v4SiiSQChopxIObCtWDkZEYpRDnllmEiICZ2SMQw0DUkA0skWt+f4XCsj7EdCV6jwQv09kZFAyjTwdGdA1ESuenPxP2+QKP/ayVgYJwpCulzkJxyrCM+DwCMmgCqeakKoYPpWTCdEEKp0XBUdgr368jrpXTbtVvPmvlVrt4o4yugUnaE6stEVaqM71EFdRNEMPaNX9GbkxovxbnwsW0tGMXOC/sD4/AFpe5Na < /latexit > For symmetric noise J = 1 , the more asymmetric the smaller J . Final weights proportional to J , ε . Putting It All Together . As a final step , to maximize the utility of the trusted data , we foster D as additional trusted training data for f ( ) . Since D contains the true labels y no prediction correction is applied . Hence , the overall loss function for data points from both D and D̃ is : l = { −A ( Ĉ ) ∑K k=1 q ( k|x ) log ( ∑K i=1 Ĉikp ( i|x ) ) −B ( Ĉ ) ∑K k=1 p ( k|x ) log q ( k|x ) , x ∈ D̃ − ∑K k=1 q ( k|x ) log p ( k|x ) , x ∈ D. ( 6 ) Figure . 3 summarises visually the training process divided into two main steps : ( i ) estimating noise corruption matrix through the first network g trained on untrusted dataset D̃ and ( ii ) training classifier f on both untrusted D̃ and trusted D through the golden symmetric loss .

Summary: The authors of the paper propose a novel loss function termed the golden symmetric loss to tackle the important problem of learning with noisy labels. The proposed loss function, in essence, involves the use of a corruption matrix to correct the regular cross-entropy loss and, at the same time, to estimate the relative weighting of the corrected cross-entropy loss and that of reverse cross-entropy loss.  A series of empirical experiments were conducted to demonstrate the effectiveness of the proposed method.