Curriculum Loss: Robust Learning and Generalization  against Label Corruption

Deep neural networks ( DNNs ) have great expressive power , which can even memorize samples with wrong labels . It is vitally important to reiterate robustness and generalization in DNNs against label corruption . To this end , this paper studies the 0-1 loss , which has a monotonic relationship with empirical adversary ( reweighted ) risk ( Hu et al. , 2018 ) . Although the 0-1 loss has some robust properties , it is difficult to optimize . To efficiently optimize the 0-1 loss while keeping its robust properties , we propose a very simple and efficient loss , i.e . curriculum loss ( CL ) . Our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses . Moreover , CL can adaptively select samples for model training . As a result , our loss can be deemed as a novel perspective of curriculum sample selection strategy , which bridges a connection between curriculum learning and robust learning . Experimental results on benchmark datasets validate the robustness of the proposed loss . 1 INTRODUCTION . Noise corruption is a common phenomenon in our daily life . For instance , noisy corrupted ( wrong ) labels may be resulted from annotating for similar objects ( Su et al. , 2012 ; Yuan et al. , 2019 ) , crawling images and labels from websites ( Hu et al. , 2017 ; Tanaka et al. , 2018 ) and creating training sets by program ( Ratner et al. , 2016 ; Khetan et al. , 2018 ) . Learning with noisy labels is thus an promising area . Deep neural networks ( DNNs ) have great expressive power ( model complexity ) to learn challenging tasks . However , DNNs also undertake a higher risk of overfitting to the data . Although many regularization techniques , such as adding regularization terms , data augmentation , weight decay , dropout and batch normalization , have been proposed , generalization is still vitally important for deep learning to fully exploit the super-expressive power . Zhang et al . ( 2017 ) show that DNNs can even fully memorize samples with incorrectly corrupted labels . Such label corruption significantly degenerates the generalization performance of deep models . This calls a lot of attention on robustness in deep learning with noisy labels . Robustness of 0-1 loss : The problem resulted from data corruption or label corruption is that test distribution is different from training distribution . Hu et al . ( 2018 ) analyzed the adversarial risk that the test distribution density is adversarially changed within a limited f -divergence ( e.g . KLdivergence ) from the training distribution density . They show that there is a monotonic relationship between the ( empirical ) risk and the ( empirical ) adversarial risk when the 0-1 loss function is used . This suggests that minimizing the empirical risk with the 0-1 loss function is equivalent to minimize the empirical adversarial risk ( worst-case risk ) . When we train a model based on the corrupted training distribution , we want our model to perform well on the clean distribution . Since we do not know the clean distribution , we want our model to perform well for the worst case estimate of the clean distribution in some constrained set . It is thus natural to employ the worst-case classification risk of the estimated clean distribution as the objective . Note that the worst-case classification risk is an upper bound of the classification risk of the true clean distribution , minimizing the worst-case risk can usually decrease the true risk . When we employ the 0-1 loss , because of the equivalence between the classification risk and the worst-case classification risk , we can directly minimize the classification risk under the corrupted training distribution instead of minimizing the worst-case classification risk . From the learning perspective , the 0-1 loss is more robust to outliers compared with an unbounded ( convex ) loss ( e.g . hinge loss ) ( Masnadi-Shirazi & Vasconcelos , 2009 ) . This is due to unbounded convex losses putting much weight on the outliers ( with a large loss value ) when minimizing the losses ( Masnadi-Shirazi & Vasconcelos , 2009 ) . If the unbounded ( convex ) loss is employed in deep network models , this becomes more prominent . Since training loss of deep networks can often be minimized to zero , outlier with a large loss has a large impact on the model . On the other hand , the 0-1 loss treats each training sample equally . Thus , each sample does not have too much influence on the model . Therefore , the model is tolerant of a small number of outliers . Although the 0-1 loss has many robust properties , its non-differentiability and zero gradients make it difficult to optimize . One possible way to alleviate this problem is to seek an upper bound of the 0-1 loss that is still efficient to optimize but tighter than conventional ( convex ) losses . Such a tighter upper bound of the 0-1 loss can reduce the influence of the noisy outliers compared with conventional ( convex ) losses . At the same time , it is easier to optimize compared with the 0-1 loss . When minimizing the upper bound surrogate , we expect that the 0-1 loss objective is also minimized . Learnability under large noise rate : The 0-1 loss can not deal with large noise rate . When the noise rate becomes large , the systematic error ( due to label corruption ) grows up and becomes not negligible . As a result , the model ’ s generalization performance will degenerate due to this systematic error . To reduce the systematic error produced by training with noisy labels , several methods have been proposed . They can be categorized into three kinds : transition matrix based method ( Sukhbaatar et al. , 2014 ; Patrini et al. , 2017 ; Goldberger & Ben-Reuven , 2017 ) , regularization based method ( Miyato et al. , 2016 ) and sample selection based method ( Jiang et al. , 2018 ; Han et al. , 2018b ) . Among them , sample selection based method is one promising direction that selects samples to reduce noisy ratio for training . These methods are based on the idea of curriculum learning ( Bengio et al. , 2009 ) which is one successful method that trains the model gradually with samples ordered in a meaningful sequence . Although they achieve success to some extents , most of these methods are heuristic based . To efficiently minimize the 0-1 loss while keeping the robust properties , we propose a novel loss that is a tighter upper bound of the 0-1 loss compared with conventional surrogate losses . Specifically , giving any base loss function l ( u ) ≥ 1 ( u < 0 ) , u ∈ R , our loss Q ( u ) satisfies∑n i=1 1 ( ui < 0 ) ≤ Q ( u ) ≤ ∑n i=1 l ( ui ) , where u = [ u1 , · · · , un ] with ui being the classification margin of ith sample , and 1 ( · ) is an indicator function . We name it as Curriculum Loss ( CL ) because our loss automatically and adaptively selects samples for training , which can be deemed as a curriculum learning paradigm . Our contributions are listed as follows : • We propose a novel loss ( i.e . curriculum loss ) for robust learning against label corruption . We prove that our CL is a tighter upper bound of 0-1 loss compared with conventional summation based surrogate loss . Moreover , CL can adaptively select samples for stagewise training , which bridges a connection between curriculum learning and robust learning . • We prove that CL can be performed by a simple and fast selection algorithm with O ( n log n ) time complexity . Moreover , our CL supports mini-batch update , which is convenient to be used as a plug-in in many deep models . • We further propose a Noise Pruned Curriculum Loss ( NPCL ) to address label corruption problem by extending CL to a more general form . Our NPCL automatically prune the estimated noisy samples during training . Moreover , NPCL is also very simple and efficient , which can be used as a plug-in in deep models as well . 2 CURRICULUM LOSS . In this section , we present the framework of our proposed Curriculum Loss ( CL ) . We begin with discussion about robustness of the 0-1 loss in Section 2.1 . We then show that our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses in Section 2.2 . A tighter bound of the 0-1 loss means that it is less sensitive to the noisy outliers , and it better preserves the robustness of the 0-1 loss with a small rate of label corruption . For a large rate of label corruption , we extend our CL to a Noise Pruned Curriculum Loss ( NPCL ) to address this issue in Section 2.3 . A simple multi-class extension and a novel soft multi-hinge loss are included in the Appendix . All the detailed proofs can be found in the Appendix as well . 2.1 ROBUSTNESS OF 0-1 LOSS AGAINST LABEL CORRUPTION . We rephrase Theorem 1 in ( Hu et al. , 2018 ) from a different perspective , which motivates us to employ the 0-1 loss for training against label corruption . Theorem 1 . ( Monotonic Relationship ) ( Hu et al . ( Hu et al. , 2018 ) ) Let p ( x , y ) and q ( x , y ) be the training and test density , respectively . Define r ( x , y ) = q ( x , y ) /p ( x , y ) and ri = r ( xi , yi ) . Let l ( ŷ , y ) = 1 ( sign ( ŷ ) 6= y ) and l ( ŷ , y ) = 1 ( argmaxk ( ŷk ) 6= y ) be 0-1 loss for binary classification and multi-class classification , respectively . Let f ( · ) be convex with f ( 1 ) = 0 . Define risk R ( θ ) , empirical risk R̂ ( θ ) , adversarial riskRadv ( θ ) and empirical adversarial risk R̂adv ( θ ) as R ( θ ) = Ep ( x , y ) [ l ( gθ ( x ) , y ) ] ( 1 ) R̂ ( θ ) = 1 n ∑n i=1 l ( gθ ( xi ) , yi ) ( 2 ) Radv ( θ ) = sup r∈Uf Ep ( x , y ) [ r ( x , y ) l ( gθ ( x ) , y ) ] ( 3 ) R̂adv ( θ ) = sup r∈Ûf 1 n ∑n i=1 ril ( gθ ( xi ) , yi ) , ( 4 ) where Uf = { r ( x , y ) ∣∣Ep ( x , y ) [ f ( r ( x , y ) ) ] ≤ δ , Ep ( x , y ) [ r ( x , y ) ] = 1 , r ( x , y ) ≥ 0 , ∀ ( x , y ) ∈ X × Y } and Ûf = { r ∣∣ 1 n ∑n i=1 f ( ri ) ≤ δ , 1 n ∑n i=1 ri = 1 , r ≥ 0 } . Then we have that If Radv ( θ1 ) < 1 , then R ( θ1 ) < R ( θ2 ) ⇐⇒ Radv ( θ1 ) < Radv ( θ2 ) . ( 5 ) If Radv ( θ1 ) = 1 , then R ( θ1 ) ≤ R ( θ2 ) ⇐⇒ Radv ( θ2 ) = 1 . ( 6 ) The same monotonic relationship holds between their empirical approximation : R̂ ( θ ) and R̂adv . Theorem 1 ( Hu et al. , 2018 ) shows that the monotonic relationship between the ( empirical ) risk and the ( empirical ) adversarial risk ( worst-case risk ) when 0-1 loss function is used . It means that minimizing ( empirical ) risk is equivalent to minimize the ( empirical ) adversarial risk ( worst-case risk ) for 0-1 loss . When we train a model based on the corrupted training distribution p ( x , y ) , we want our model to perform well on the clean distribution q ( x , y ) . Since we do not know the clean distribution q , we want our model to perform well for the worst-case estimate of the clean distribution , with the assumption that the f -divergence between the corrupted distribution p and the clean distribution q is bounded by δ . Note that the underlying clean distribution is fixed but unknown , given the corrupted training distribution , the smallest δ that bounds the divergence between the corrupted distribution and clean distribution measures the intrinsic difficulty of the corruption , and it is also fixed and unknown . The corresponding worst-case distribution w.r.t the smallest δ is an estimate of the true clean distribution , and this worst-case risk upper bounds the risk of the true clean distribution . In addition , this bound is tighter than the other worst-case risks w.r.t larger δ . It is natural to use this upper bound as the objective for robust learning . When we use 0-1 loss ( that is commonly employed for evaluation ) , because of the equivalence of the risk and the worst-case risk , we can directly minimize risk under training distribution p instead of directly minimizing the worst-case risk ( i.e. , the upper bound ) . Moreover , this enables us to minimize the upper bound without knowing the true δ beforehand . When the true δ is small , i.e. , the corruption of the training data is not heavy , the upper bound is not too pessimistic . Usually , minimizing the upper bound can decrease the true risk under clean distribution . Particularly , when the clean distribution coincides with the worst-case estimate w.r.t the smallest δ , minimizing the risk under the corrupted training distribution leads to the same minimizer as minimizing the risk under the clean distribution .

Summary: This paper proposes a new loss function: curriculum loss, which is a meta-loss function that we can still specify an existing surrogate loss to use this loss function. This meta-loss function guarantees to be tighter than using a traditional pointwise-sum loss function as used in the empirical risk minimization framework. Intuitively, the proposed CL loss embed the sample selection process in the objective function. The authors suggest that it is robust against label corruption because it is tighter and provided promising experimental results.