Robust Loss Functions for Complementary Labels Learning

In ordinary-label learning , the correct label is given to each training sample . 1 Similarly , a complementary label is also provided for each training sample in 2 complementary-label learning . A complementary label indicates a class that the 3 example does not belong to . Robust learning of classifiers has been investi- 4 gated from many viewpoints under label noise , but little attention has been paid 5 to complementary-label learning . In this paper , we present a new algorithm of 6 complementary-label learning with the robustness of loss function . We also pro- 7 vide two sufficient conditions on a loss function so that the minimizer of the risk 8 for complementary labels is theoretically guaranteed to be consistent with the min- 9 imizer of the risk for ordinary labels . Finally , the empirical results validate our 10 method ’ s superiority to current state-of-the-art techniques . Especially in cifar10 , 11 our algorithm achieves a much higher test accuracy than the gradient ascent algo- 12 rithm , and the parameters of our model are less than half of the ResNet-34 they 13 used . 14 1 INSTRUCTION 15 . Deep neural networks have exhibited excellent performance in many real-applications . Yet , their 16 supper performance is based on the correctly labeled large-scale training set . However , labeling 17 such a large-scale dataset is time-consuming and expensive . For example , the crowd-workers need 18 to select the correct label for a sample from 100 labels for CIFAR100 . To migrate this problem , 19 reachers have proposed many solutions to learn from weak-supervision : Noise-label learning Li 20 et al . ( 2017 ) ; Hu et al . ( 2019 ) ; Lee et al . ( 2018 ) ; Xia et al . ( 2019 ) , semi-supervised learning Zhai 21 et al . ( 2019 ) ; Berthelot et al . ( 2019 ) ; Rasmus et al . ( 2015 ) ; Miyato et al . ( 2019 ) ; Sakai et al . ( 2017 ) , 22 similar-unlabeled learning Tanha ( 2019 ) ; Bao et al . ( 2018 ) ; Zelikovitz & Hirsh ( 2000 ) , unlabeled- 23 unlabeled learning Lu et al . ( 2018 ) ; Chen et al . ( 2020a ; b ) , positive-unlabeled learning Elkan & Noto 24 ( 2008 ) ; du Plessis et al . ( 2014 ) ; Kiryo et al . ( 2017 ) , contrast learning Chen et al . ( 2020a ; b ) , partial 25 label learning Cour et al . ( 2011 ) ; Feng & An ( 2018 ) ; Wu & Zhang ( 2018 ) and others . 26 We investigate complementary-label learning Ishida et al . ( 2017 ) in this paper . A complementary 27 Label is only indicating that the class label of a sample is incorrect . In the view of label noise , 28 complementary labels can also be viewed as noise labels but without any true labels in the training 29 set . Our task is to learn a classifier from the given complementary labels , predicting a correct label 30 for a given sample . Collecting complementary labels is much easier and efficient than choosing a 31 true class from many candidate classes precisely . For example , the label-system uniformly chooses a 32 label for a sample . It has a probability of 1k to be ordinary-label but k−1 k to be complementary-label . 33 Moreover , another potential application of complementary-label is data privacy . For example , on 34 some privacy issues , it is much easier to collect complementary-label than ordinary-label . 35 Robust learning of classifiers has been investigated from many viewpoints in the presence of label 36 noise Ghosh et al . ( 2017 ) , but little attention paid to complementary-label learning . We call a loss 37 function robust if the minimizer of risk under that loss function with complementary labels would be 38 the same as that with ordinary labels . The robustness of risk minimization relies on the loss function 39 used in the training set . 40 This paper presents a general risk formulation that category cross-entropy loss ( CCE ) can be used to 41 learn with complementary labels and achieve robustness . We then offer some innovative analytical 42 results on robust loss functions under complementary labels . Having robustness of risk minimization 43 helps select the best hyper-parameter by empirical risk since there are no ordinary labels in the44 validation set . We conclude two sufficient conditions on a loss function to be robust for learning45 with complementary labels . We then explore some popular loss functions used for ordinary-label46 learning , such as CCE , Mean square error ( MSE ) and Mean absolute error ( MAE ) , and show that47 CCE and MAE satisfy our sufficient conditions . Finally , we present a learning algorithm for learning48 with complementary labels , named exclusion algorithm . The empirical results well demonstrate the49 advantage of the theoretical results we addressed and verify our algorithm ’ s superiority to the current50 state-of-the-art methods . The contribution of this paper can be summarized as:51 • We present a general risk formulation that can be view as a framework to employing a52 loss function that satisfies our robustness sufficient condition to learn from complementary53 labels.54 • We conclude two sufficient conditions on a loss function to be robust for learning with55 complementary labels.56 • We prove that the minimizer of the risk for complementary labels is theoretically guaran-57 teed to be consistent with the minimizer of the risk for ordinary labels.58 • The empirical results validate the superiority of our method to current state-of-the-art meth-59 ods.60 2 RELATED WORKS61 Complementary-label refers to that the pattern does not belong to the given label . Learning from62 complementary labels is a new topic in supervised-learning . It was first proposed by Ishida et al.63 ( 2017 ) . They conduct such an idea to try to deal with time-consuming and expensive to tag a large-64 scale dataset.65 In their early work Ishida et al . ( 2017 ) , they assume the complementary labels are the same prob-66 ability to be selected for a sample . And then , based on the ordinary one-versus-all ( OVA ) and67 pairwise-comparison ( PC ) multi-class loss functions Zhang ( 2004 ) proposed a modifying loss for68 learning with complementary labels.69 Even though they provided theoretical analysis with a statistical consistency guarantee , the loss70 function met a sturdy restriction that needs to be symmetric ( ` ( z ) + ` ( −z ) = 1 ) . Such a severe71 limitation allows only the OVA and PC loss functions with symmetric non-convex binary losses.72 However , the categorical cross-entropy loss widely used in the deep learning domain , can not be73 employed by the two losses they defined.74 Later , Yu et al . ( 2018a ) assume there are some biased amongst the complementary labels and75 presents a different formulation for biased complementary labels by using the forward loss cor-76 rection technique Patrini et al . ( 2017 ) to modify traditional loss functions . Their suggested risk77 estimator is not necessarily unbiased and proved that learning with complementary labels can the-78 oretically converge to the optimal classifier learned from ordinary labels based on the estimated79 transition matrix . However , the key to the forward loss correction technique is to evaluate the tran-80 sition matrix correctly . Hence , one will need to assess the transition matrix beforehand , which is81 relatively tricky without strong assumptions . Moreover , in such a setup , it restricts a small com-82 plementary label space to provide more information . Thus , it is necessary to encourage the worker83 to provide more challenging complementary labels , for example , by giving higher rewards to the84 specific classes . Otherwise , the complementary label given by the worker may be too evident and85 uninformative . For example , class three and class five are not class one evidently but is uninforma-86 tive . This paper focuses on the uniform ( symmetric ) assumption and study random distribution as a87 biased assumption ( asymmetric or non-uniform ) .88 Based on the uniform assumption , Ishida et al . ( 2019 ) proposed an unbiased estimator with a general89 loss function for complementary labels . It can make any loss functions available for use , not only90 soft-max cross-entropy loss function , but other loss functions can also be utilized . Their new frame-91 work is a generalization of previous complementary-label learning Ishida et al . ( 2017 ) . However,92 their proposed unbiased risk estimator has an issue that the classification risk can attain negative93 values after learning , leading to overfitting Ishida et al . ( 2019 ) . They then offered a non-negative94 correction to the original unbiased risk estimator to improve their estimator , which is no longer95 guaranteed to be an unbiased risk estimator . In this paper , our proposed risk estimator is also not 96 unbiased , but the minimizer of the risk for complementary labels is theoretically guaranteed to be 97 consistent with the minimizer of the risk for ordinary labels , both uniform and non-uniform . 98 3 PRELIMINARIES 99 . 3.1 LEARNING WITH ORDINARY LABELS 100 In the context of learning with ordinary labels , let X ⊂ Rd be the feature space and Y = { 1 , · · · , k } 101 be the class labels . A multi-class loss function is a map : L ( fθ ( x ) , y ) : X × Y → R+ . A classifier 102 can be presented as : 103 h ( x ) = arg max i∈ [ k ] f ( i ) θ ( x ) , ( 1 ) where fθ ( x ) = ( f ( 1 ) θ ( x ) , · · · , f ( k ) θ ( x ) ) , θ is the set of parameters in the CNN network , f ( i ) θ ( x ) is 104 the probability prediction for the corresponding class i . Even though h ( x ) is the final classifier , we 105 use notation of calling fθ ( x ) itself as the classifier . Given dataset S = { ( xi , yi ) } Ni , together with a 106 loss function L , ∀fθ ∈ F ( F is the function space for searching ) , L-risk is defined as : 107 RSL ( fθ ) = ED [ L ( fθ ( x ) , y ) ] = ES [ L ( fθ ( x ) , y ) ] , ( 2 ) Some popular multi-class loss functions are CCE , MAE , MSE . Specifically , 108 ` ( fθ ( x ) , y ) = ` ( u , y ) = ∑k i=1 e ( i ) y log 1 µy = log 1µy CCE , ‖ey − u‖1 = 2− 2µy MAE , ‖ey − u‖22 = ‖u‖ 2 2 + 1− 2µy MSE , ( 3 ) where u = fθ ( x ) = ( µ1 , · · · , µk ) , and ey is a one-hot vector that the y-th component equals to 1 , 109 others are 0 . The goal of multi-class classification is to learn a classifier fθ ( x ) that minimize the 110 classification riskRSL with multi-class loss L . 111 3.2 LEARNING WITH COMPLEMENTARY LABELS 112 In contrast to the ordinary-label learning , the complementary-label ( CL ) dataset contains only labels 113 indicating that the class label of a sample is incorrect . Corresponding to the ordinary labels dataset 114 S , the independent and identically distributed ( i.i.d . ) complementary labels dataset denoted as : 115 S̄ = { ( x , ȳ ) } Ni , ( 4 ) where N is the size of the dataset S̄ , and ȳ represents that pattern x does not belong to class-ȳ . 116 The general labels ’ distribution of dataset S̄ is as : 117 P ( ȳ|y ) = 0 p12 . . . p1k p21 0 . . . p2k ... ... . . . ... pk1 . . . pk ( k−1 ) 0 k×k , ( 5 ) where pij denotes that the probability of the i-th class ’ s pattern x labeled as j , ∑k j=1 pij = 118 1 , pij 6=0 , j 6= i. Supposing that the label system uniformly select a label from { 1 , · · · , k } \ { y } 119 for each sample x , then the Eq . ( 5 ) becomes 120 P ( ȳ|y ) = 0 1k−1 . . . 1 k−1 1 k−1 0 . . . 1 k−1 ... ... . . . ... 1 k−1 . . . 1 k−1 0 k×k . ( 6 ) Yu et al . ( 2018b ) make a strong assumption that there are some bias in Eq . ( 5 ) , while Ishida 121 et al . ( 2017 ; 2019 ) focus on the assumption of Eq . ( 6 ) . In this paper , we study both kinds of distri- 122 bution . 123 4 METHODOLOGY124 In this section , we firstly propose a general risk formulation for leaning with complementary labels.125 And then prove that some loss functions designed for the ordinary labels learning are robust to126 complementary labels with our risk formulation , such as categorical cross-entropy loss and mean127 absolute error.128 4.1 GENERAL RISK FORMULATION129 The goal of learning with complementary labels is to learn a classifier that predicts a correct label for130 any sample drawn from the same distribution . Because there are not ordinary labels for the model,131 we need to design a loss function or model for learning with complementary labels . The key to learn-132 ing a classifier for ordinary label learning is to maximize the true label ’ s predict-probability . One133 intuitive way to maximize the true label ’ s predict-probability is to minimize the predict-probability134 of complementary labels . In this paper , with little abuse of notation , we let135 u = fθ ( x ) = ( µ1 , · · · , µk ) v = 1− fθ ( x ) = ( 1− µ1 , · · · , 1− µk ) . ( 7 ) Definition 1 . ( CL-loss ) Together with loss function ` designed for the ordinary-label learning , the136 loss for learning with complementary-label is as:137 ¯̀ ( fθ ( x ) , ȳ ) = ¯̀ ( u , ȳ ) = ` ( v , ȳ ) . ( 8 ) 4.2 THEORETICAL RESULTS138 Definition 2 . ( Robust loss ) In the framework of risk minimization , a loss function is called robust139 loss function if minimizer of risk with complementary labels would be the same as with ordinary140 labels , i.e.,141 RS̄¯̀ ( fθ∗ ) −RS̄¯̀ ( fθ ) ≤ 0⇒ RS ` ( fθ∗ ) −RS ` ( fθ ) ≤ 0 . ( 9 ) Theorem 1 . Together with ` , ¯̀ is a robust loss function for learning with complementary labels , if ¯̀142 satisfies:143 ∂ ¯̀ ( u , ȳ ) ∂µȳ > 0 , ∂ ¯̀ ( u , ȳ ) ∂µi = 0 , ∀i ∈ { 1 , · · · , k } \ { ȳ } . ( 10 ) Note that , in Eq . 10 , it means that ¯̀ is a monotone increasing loss function only on u ( ȳ ) .144 Proof . Recall that for any fθ , and any ` ,145 RS ` ( fθ ) = E ( x , y ) [ ` ( fθ ( x ) , y ) ] = 1 |S| ∑ ( x , y ) ∈S ` ( fθ ( x ) , y ) . ( 11 ) For any complementary-label distribution in Eq . ( 5 ) , and any loss function ` , we have146 RS̄¯̀ ( fθ ) = E ( x , ȳ ) [ ¯̀ ( fθ ( x ) , ȳ ) ] = 1 |S̄| k∑ i=1 ∑ x∈Si k∑ j 6=i pij ¯̀ ( fθ ( x ) , j ) , ( 12 ) where pij is the component of complementary labels distribution matrix P , S1 ∪ · · · ∪ Sk = S .147 Supposing that fθ∗ is the optimal classifier learns from the complementary labels , and ∀f ∈ F ,148 where F is the function space for searching , we have149 RS̄¯̀ ( fθ∗ ) −RS̄¯̀ ( fθ ) = 1 |S̄| k∑ i=1 ∑ x∈Si k∑ j 6=i pij ( ¯̀ ( fθ∗ ( x ) , j ) − ¯̀ ( fθ ( x ) , j ) ) ≤ 0 , ( 13 ) where pij 6= 0 . If ∃x ′ ∈ S̄ , s.t. , ¯̀ ( fθ∗ ( x ′ ) , ȳ ) > ¯̀ ( fθ ( x ′ ) , ȳ ) , let fθ′ satisfying150 fθ′ ( x ) = { fθ∗ ( x ) x ∈ S̄ \ { x ′ } , fθ ( x ) x = x ′ , ( 14 ) then according to Eq . 12 and 13 , RS̄¯̀ ( fθ′ ) < RS̄¯̀ ( fθ∗ ) , fθ∗ is not the optimal classifier . This 151 contradicts the hypothesize that fθ∗ is the optimal classifier . 152 Thus , ∀ȳ ∈ { 1 , · · · , k } \ { y } , we have 153 ¯̀ ( fθ∗ ( x ) , ȳ ) ≤ ¯̀ ( fθ ( x ) , ȳ ) . ( 15 ) According to Eq . ( 10 ) , ¯̀ is a monotone increasing loss function only on u ( ȳ ) , then we have 154 ∀ȳ ∈ { 1 , · · · , k } \ { y } , f ( ȳ ) θ∗ ( x ) ≤ f ( ȳ ) θ ( x ) . ( 16 ) Thus , 155 f ( y ) θ∗ ( x ) ≥ f ( y ) θ ( x ) , f ( y ) θ ( x ) = 1−∑ ȳ 6=y f ( ȳ ) θ ( x ) ( 17 ) and then , 156 ` ( fθ∗ ( x ) , y ) ≤ ` ( fθ ( x ) , y ) , ( 18 ) thus , 157 RS ` ( fθ∗ ) −RS ` ( fθ ) ≤ 0 . ( 19 ) 158 Theorem 2 . Together with ` , ¯̀ is a robust loss function for learning with complementary labels 159 under symmetric distribution or uniform distribution , if ¯̀satisfies : 160 ∂ ¯̀ ( u , ȳ ) ∂µȳ > 0 , k∑ i=1 ¯̀ ( u , i ) = C , ( C is a constant ) . ( 20 ) It should be noted that , in Eq . 20 , it means that ¯̀ is a symmetric loss ( ∑ ` ( u , i ) = C ) , and ¯̀ is a 161 monotone increasing loss function on any ȳ . 162 Proof . For any complementary-label distribution in Eq . ( 6 ) , and any loss function ` , we have 163 RS̄¯̀ ( fθ ) = E ( x , ȳ ) [ ¯̀ ( fθ ( x ) , ȳ ) ] = 1 |S̄| k∑ i=1 ∑ x∈Si k∑ j 6=i 1 k − 1 ¯̀ ( fθ ( x ) , j ) = 1 |S̄| k∑ i=1 ∑ x∈Si 1 k − 1 ( C − ¯̀ ( fθ ( x ) , i ) ) = C k − 1 −RS¯̀ ( fθ ) , ( 21 ) where S1 ∪ · · · ∪ Sk = S . 164 Supposing that fθ∗ is the optimal classifier learns from the complementary labels , and ∀f ∈ F , 165 where F is the function space for searching , we have 166 RS̄¯̀ ( fθ∗ ) −RS̄¯̀ ( fθ ) = RS¯̀ ( fθ ) −RS¯̀ ( fθ∗ ) ≤ 0 , ( 22 ) According to the first constraint in Eq . ( 20 ) , we then have 167 ¯̀ ( fθ ( x ) , y ) ≤ ¯̀ ( fθ∗ ( x ) , y ) , ( f ( y ) θ ( x ) ≤ f ( y ) θ∗ ( x ) ) ( 23 ) and then , 168 ` ( fθ∗ ( x ) , y ) ≤ ` ( fθ ( x ) , y ) , ( 24 ) thus , 169 RS ` ( fθ∗ ) −RS ` ( fθ ) ≤ 0 . ( 25 ) 170 Algorithm 1 Learning from complementary labels by exclusion

Summary: This paper studied a new problem, that is, learning from complementary labels.  The goal is to predict a correct label for a given sample when only given complementary labels. On the basis of the ordinary-label learning, the authors defined "robust loss functions" for complementary-label learning:  a a loss function is called robust  loss function if minimizer of risk with complementary labels would be the same as with ordinary  labels. Then, they provided  two sufficient conditions for the robust loss function and a exclusion algorithm is provided for prediction. Experimental results show that the proposed method outperforms other methods in several datasets.