Robust Loss Functions for Complementary Labels Learning

In ordinary-label learning , the correct label is given to each training sample . 1 Similarly , a complementary label is also provided for each training sample in 2 complementary-label learning . A complementary label indicates a class that the 3 example does not belong to . Robust learning of classifiers has been investi- 4 gated from many viewpoints under label noise , but little attention has been paid 5 to complementary-label learning . In this paper , we present a new algorithm of 6 complementary-label learning with the robustness of loss function . We also pro- 7 vide two sufficient conditions on a loss function so that the minimizer of the risk 8 for complementary labels is theoretically guaranteed to be consistent with the min- 9 imizer of the risk for ordinary labels . Finally , the empirical results validate our 10 method ’ s superiority to current state-of-the-art techniques . Especially in cifar10 , 11 our algorithm achieves a much higher test accuracy than the gradient ascent algo- 12 rithm , and the parameters of our model are less than half of the ResNet-34 they 13 used . 14 1 INSTRUCTION 15 . Deep neural networks have exhibited excellent performance in many real-applications . Yet , their 16 supper performance is based on the correctly labeled large-scale training set . However , labeling 17 such a large-scale dataset is time-consuming and expensive . For example , the crowd-workers need 18 to select the correct label for a sample from 100 labels for CIFAR100 . To migrate this problem , 19 reachers have proposed many solutions to learn from weak-supervision : Noise-label learning Li 20 et al . ( 2017 ) ; Hu et al . ( 2019 ) ; Lee et al . ( 2018 ) ; Xia et al . ( 2019 ) , semi-supervised learning Zhai 21 et al . ( 2019 ) ; Berthelot et al . ( 2019 ) ; Rasmus et al . ( 2015 ) ; Miyato et al . ( 2019 ) ; Sakai et al . ( 2017 ) , 22 similar-unlabeled learning Tanha ( 2019 ) ; Bao et al . ( 2018 ) ; Zelikovitz & Hirsh ( 2000 ) , unlabeled- 23 unlabeled learning Lu et al . ( 2018 ) ; Chen et al . ( 2020a ; b ) , positive-unlabeled learning Elkan & Noto 24 ( 2008 ) ; du Plessis et al . ( 2014 ) ; Kiryo et al . ( 2017 ) , contrast learning Chen et al . ( 2020a ; b ) , partial 25 label learning Cour et al . ( 2011 ) ; Feng & An ( 2018 ) ; Wu & Zhang ( 2018 ) and others . 26 We investigate complementary-label learning Ishida et al . ( 2017 ) in this paper . A complementary 27 Label is only indicating that the class label of a sample is incorrect . In the view of label noise , 28 complementary labels can also be viewed as noise labels but without any true labels in the training 29 set . Our task is to learn a classifier from the given complementary labels , predicting a correct label 30 for a given sample . Collecting complementary labels is much easier and efficient than choosing a 31 true class from many candidate classes precisely . For example , the label-system uniformly chooses a 32 label for a sample . It has a probability of 1k to be ordinary-label but k−1 k to be complementary-label . 33 Moreover , another potential application of complementary-label is data privacy . For example , on 34 some privacy issues , it is much easier to collect complementary-label than ordinary-label . 35 Robust learning of classifiers has been investigated from many viewpoints in the presence of label 36 noise Ghosh et al . ( 2017 ) , but little attention paid to complementary-label learning . We call a loss 37 function robust if the minimizer of risk under that loss function with complementary labels would be 38 the same as that with ordinary labels . The robustness of risk minimization relies on the loss function 39 used in the training set . 40 This paper presents a general risk formulation that category cross-entropy loss ( CCE ) can be used to 41 learn with complementary labels and achieve robustness . We then offer some innovative analytical 42 results on robust loss functions under complementary labels . Having robustness of risk minimization 43 helps select the best hyper-parameter by empirical risk since there are no ordinary labels in the44 validation set . We conclude two sufficient conditions on a loss function to be robust for learning45 with complementary labels . We then explore some popular loss functions used for ordinary-label46 learning , such as CCE , Mean square error ( MSE ) and Mean absolute error ( MAE ) , and show that47 CCE and MAE satisfy our sufficient conditions . Finally , we present a learning algorithm for learning48 with complementary labels , named exclusion algorithm . The empirical results well demonstrate the49 advantage of the theoretical results we addressed and verify our algorithm ’ s superiority to the current50 state-of-the-art methods . The contribution of this paper can be summarized as:51 • We present a general risk formulation that can be view as a framework to employing a52 loss function that satisfies our robustness sufficient condition to learn from complementary53 labels.54 • We conclude two sufficient conditions on a loss function to be robust for learning with55 complementary labels.56 • We prove that the minimizer of the risk for complementary labels is theoretically guaran-57 teed to be consistent with the minimizer of the risk for ordinary labels.58 • The empirical results validate the superiority of our method to current state-of-the-art meth-59 ods.60 2 RELATED WORKS61 Complementary-label refers to that the pattern does not belong to the given label . Learning from62 complementary labels is a new topic in supervised-learning . It was first proposed by Ishida et al.63 ( 2017 ) . They conduct such an idea to try to deal with time-consuming and expensive to tag a large-64 scale dataset.65 In their early work Ishida et al . ( 2017 ) , they assume the complementary labels are the same prob-66 ability to be selected for a sample . And then , based on the ordinary one-versus-all ( OVA ) and67 pairwise-comparison ( PC ) multi-class loss functions Zhang ( 2004 ) proposed a modifying loss for68 learning with complementary labels.69 Even though they provided theoretical analysis with a statistical consistency guarantee , the loss70 function met a sturdy restriction that needs to be symmetric ( ` ( z ) + ` ( −z ) = 1 ) . Such a severe71 limitation allows only the OVA and PC loss functions with symmetric non-convex binary losses.72 However , the categorical cross-entropy loss widely used in the deep learning domain , can not be73 employed by the two losses they defined.74 Later , Yu et al . ( 2018a ) assume there are some biased amongst the complementary labels and75 presents a different formulation for biased complementary labels by using the forward loss cor-76 rection technique Patrini et al . ( 2017 ) to modify traditional loss functions . Their suggested risk77 estimator is not necessarily unbiased and proved that learning with complementary labels can the-78 oretically converge to the optimal classifier learned from ordinary labels based on the estimated79 transition matrix . However , the key to the forward loss correction technique is to evaluate the tran-80 sition matrix correctly . Hence , one will need to assess the transition matrix beforehand , which is81 relatively tricky without strong assumptions . Moreover , in such a setup , it restricts a small com-82 plementary label space to provide more information . Thus , it is necessary to encourage the worker83 to provide more challenging complementary labels , for example , by giving higher rewards to the84 specific classes . Otherwise , the complementary label given by the worker may be too evident and85 uninformative . For example , class three and class five are not class one evidently but is uninforma-86 tive . This paper focuses on the uniform ( symmetric ) assumption and study random distribution as a87 biased assumption ( asymmetric or non-uniform ) .88 Based on the uniform assumption , Ishida et al . ( 2019 ) proposed an unbiased estimator with a general89 loss function for complementary labels . It can make any loss functions available for use , not only90 soft-max cross-entropy loss function , but other loss functions can also be utilized . Their new frame-91 work is a generalization of previous complementary-label learning Ishida et al . ( 2017 ) . However,92 their proposed unbiased risk estimator has an issue that the classification risk can attain negative93 values after learning , leading to overfitting Ishida et al . ( 2019 ) . They then offered a non-negative94 correction to the original unbiased risk estimator to improve their estimator , which is no longer95 guaranteed to be an unbiased risk estimator . In this paper , our proposed risk estimator is also not 96 unbiased , but the minimizer of the risk for complementary labels is theoretically guaranteed to be 97 consistent with the minimizer of the risk for ordinary labels , both uniform and non-uniform . 98 3 PRELIMINARIES 99 . 3.1 LEARNING WITH ORDINARY LABELS 100 In the context of learning with ordinary labels , let X ⊂ Rd be the feature space and Y = { 1 , · · · , k } 101 be the class labels . A multi-class loss function is a map : L ( fθ ( x ) , y ) : X × Y → R+ . A classifier 102 can be presented as : 103 h ( x ) = arg max i∈ [ k ] f ( i ) θ ( x ) , ( 1 ) where fθ ( x ) = ( f ( 1 ) θ ( x ) , · · · , f ( k ) θ ( x ) ) , θ is the set of parameters in the CNN network , f ( i ) θ ( x ) is 104 the probability prediction for the corresponding class i . Even though h ( x ) is the final classifier , we 105 use notation of calling fθ ( x ) itself as the classifier . Given dataset S = { ( xi , yi ) } Ni , together with a 106 loss function L , ∀fθ ∈ F ( F is the function space for searching ) , L-risk is defined as : 107 RSL ( fθ ) = ED [ L ( fθ ( x ) , y ) ] = ES [ L ( fθ ( x ) , y ) ] , ( 2 ) Some popular multi-class loss functions are CCE , MAE , MSE . Specifically , 108 ` ( fθ ( x ) , y ) = ` ( u , y ) = ∑k i=1 e ( i ) y log 1 µy = log 1µy CCE , ‖ey − u‖1 = 2− 2µy MAE , ‖ey − u‖22 = ‖u‖ 2 2 + 1− 2µy MSE , ( 3 ) where u = fθ ( x ) = ( µ1 , · · · , µk ) , and ey is a one-hot vector that the y-th component equals to 1 , 109 others are 0 . The goal of multi-class classification is to learn a classifier fθ ( x ) that minimize the 110 classification riskRSL with multi-class loss L . 111 3.2 LEARNING WITH COMPLEMENTARY LABELS 112 In contrast to the ordinary-label learning , the complementary-label ( CL ) dataset contains only labels 113 indicating that the class label of a sample is incorrect . Corresponding to the ordinary labels dataset 114 S , the independent and identically distributed ( i.i.d . ) complementary labels dataset denoted as : 115 S̄ = { ( x , ȳ ) } Ni , ( 4 ) where N is the size of the dataset S̄ , and ȳ represents that pattern x does not belong to class-ȳ . 116 The general labels ’ distribution of dataset S̄ is as : 117 P ( ȳ|y ) = 0 p12 . . . p1k p21 0 . . . p2k ... ... . . . ... pk1 . . . pk ( k−1 ) 0 k×k , ( 5 ) where pij denotes that the probability of the i-th class ’ s pattern x labeled as j , ∑k j=1 pij = 118 1 , pij 6=0 , j 6= i. Supposing that the label system uniformly select a label from { 1 , · · · , k } \ { y } 119 for each sample x , then the Eq . ( 5 ) becomes 120 P ( ȳ|y ) = 0 1k−1 . . . 1 k−1 1 k−1 0 . . . 1 k−1 ... ... . . . ... 1 k−1 . . . 1 k−1 0 k×k . ( 6 ) Yu et al . ( 2018b ) make a strong assumption that there are some bias in Eq . ( 5 ) , while Ishida 121 et al . ( 2017 ; 2019 ) focus on the assumption of Eq . ( 6 ) . In this paper , we study both kinds of distri- 122 bution . 123 4 METHODOLOGY124 In this section , we firstly propose a general risk formulation for leaning with complementary labels.125 And then prove that some loss functions designed for the ordinary labels learning are robust to126 complementary labels with our risk formulation , such as categorical cross-entropy loss and mean127 absolute error.128 4.1 GENERAL RISK FORMULATION129 The goal of learning with complementary labels is to learn a classifier that predicts a correct label for130 any sample drawn from the same distribution . Because there are not ordinary labels for the model,131 we need to design a loss function or model for learning with complementary labels . The key to learn-132 ing a classifier for ordinary label learning is to maximize the true label ’ s predict-probability . One133 intuitive way to maximize the true label ’ s predict-probability is to minimize the predict-probability134 of complementary labels . In this paper , with little abuse of notation , we let135 u = fθ ( x ) = ( µ1 , · · · , µk ) v = 1− fθ ( x ) = ( 1− µ1 , · · · , 1− µk ) . ( 7 ) Definition 1 . ( CL-loss ) Together with loss function ` designed for the ordinary-label learning , the136 loss for learning with complementary-label is as:137 ¯̀ ( fθ ( x ) , ȳ ) = ¯̀ ( u , ȳ ) = ` ( v , ȳ ) . ( 8 ) 4.2 THEORETICAL RESULTS138 Definition 2 . ( Robust loss ) In the framework of risk minimization , a loss function is called robust139 loss function if minimizer of risk with complementary labels would be the same as with ordinary140 labels , i.e.,141 RS̄¯̀ ( fθ∗ ) −RS̄¯̀ ( fθ ) ≤ 0⇒ RS ` ( fθ∗ ) −RS ` ( fθ ) ≤ 0 . ( 9 ) Theorem 1 . Together with ` , ¯̀ is a robust loss function for learning with complementary labels , if ¯̀142 satisfies:143 ∂ ¯̀ ( u , ȳ ) ∂µȳ > 0 , ∂ ¯̀ ( u , ȳ ) ∂µi = 0 , ∀i ∈ { 1 , · · · , k } \ { ȳ } . ( 10 ) Note that , in Eq . 10 , it means that ¯̀ is a monotone increasing loss function only on u ( ȳ ) .144 Proof . Recall that for any fθ , and any ` ,145 RS ` ( fθ ) = E ( x , y ) [ ` ( fθ ( x ) , y ) ] = 1 |S| ∑ ( x , y ) ∈S ` ( fθ ( x ) , y ) . ( 11 ) For any complementary-label distribution in Eq . ( 5 ) , and any loss function ` , we have146 RS̄¯̀ ( fθ ) = E ( x , ȳ ) [ ¯̀ ( fθ ( x ) , ȳ ) ] = 1 |S̄| k∑ i=1 ∑ x∈Si k∑ j 6=i pij ¯̀ ( fθ ( x ) , j ) , ( 12 ) where pij is the component of complementary labels distribution matrix P , S1 ∪ · · · ∪ Sk = S .147 Supposing that fθ∗ is the optimal classifier learns from the complementary labels , and ∀f ∈ F ,148 where F is the function space for searching , we have149 RS̄¯̀ ( fθ∗ ) −RS̄¯̀ ( fθ ) = 1 |S̄| k∑ i=1 ∑ x∈Si k∑ j 6=i pij ( ¯̀ ( fθ∗ ( x ) , j ) − ¯̀ ( fθ ( x ) , j ) ) ≤ 0 , ( 13 ) where pij 6= 0 . If ∃x ′ ∈ S̄ , s.t. , ¯̀ ( fθ∗ ( x ′ ) , ȳ ) > ¯̀ ( fθ ( x ′ ) , ȳ ) , let fθ′ satisfying150 fθ′ ( x ) = { fθ∗ ( x ) x ∈ S̄ \ { x ′ } , fθ ( x ) x = x ′ , ( 14 ) then according to Eq . 12 and 13 , RS̄¯̀ ( fθ′ ) < RS̄¯̀ ( fθ∗ ) , fθ∗ is not the optimal classifier . This 151 contradicts the hypothesize that fθ∗ is the optimal classifier . 152 Thus , ∀ȳ ∈ { 1 , · · · , k } \ { y } , we have 153 ¯̀ ( fθ∗ ( x ) , ȳ ) ≤ ¯̀ ( fθ ( x ) , ȳ ) . ( 15 ) According to Eq . ( 10 ) , ¯̀ is a monotone increasing loss function only on u ( ȳ ) , then we have 154 ∀ȳ ∈ { 1 , · · · , k } \ { y } , f ( ȳ ) θ∗ ( x ) ≤ f ( ȳ ) θ ( x ) . ( 16 ) Thus , 155 f ( y ) θ∗ ( x ) ≥ f ( y ) θ ( x ) , f ( y ) θ ( x ) = 1−∑ ȳ 6=y f ( ȳ ) θ ( x ) ( 17 ) and then , 156 ` ( fθ∗ ( x ) , y ) ≤ ` ( fθ ( x ) , y ) , ( 18 ) thus , 157 RS ` ( fθ∗ ) −RS ` ( fθ ) ≤ 0 . ( 19 ) 158 Theorem 2 . Together with ` , ¯̀ is a robust loss function for learning with complementary labels 159 under symmetric distribution or uniform distribution , if ¯̀satisfies : 160 ∂ ¯̀ ( u , ȳ ) ∂µȳ > 0 , k∑ i=1 ¯̀ ( u , i ) = C , ( C is a constant ) . ( 20 ) It should be noted that , in Eq . 20 , it means that ¯̀ is a symmetric loss ( ∑ ` ( u , i ) = C ) , and ¯̀ is a 161 monotone increasing loss function on any ȳ . 162 Proof . For any complementary-label distribution in Eq . ( 6 ) , and any loss function ` , we have 163 RS̄¯̀ ( fθ ) = E ( x , ȳ ) [ ¯̀ ( fθ ( x ) , ȳ ) ] = 1 |S̄| k∑ i=1 ∑ x∈Si k∑ j 6=i 1 k − 1 ¯̀ ( fθ ( x ) , j ) = 1 |S̄| k∑ i=1 ∑ x∈Si 1 k − 1 ( C − ¯̀ ( fθ ( x ) , i ) ) = C k − 1 −RS¯̀ ( fθ ) , ( 21 ) where S1 ∪ · · · ∪ Sk = S . 164 Supposing that fθ∗ is the optimal classifier learns from the complementary labels , and ∀f ∈ F , 165 where F is the function space for searching , we have 166 RS̄¯̀ ( fθ∗ ) −RS̄¯̀ ( fθ ) = RS¯̀ ( fθ ) −RS¯̀ ( fθ∗ ) ≤ 0 , ( 22 ) According to the first constraint in Eq . ( 20 ) , we then have 167 ¯̀ ( fθ ( x ) , y ) ≤ ¯̀ ( fθ∗ ( x ) , y ) , ( f ( y ) θ ( x ) ≤ f ( y ) θ∗ ( x ) ) ( 23 ) and then , 168 ` ( fθ∗ ( x ) , y ) ≤ ` ( fθ ( x ) , y ) , ( 24 ) thus , 169 RS ` ( fθ∗ ) −RS ` ( fθ ) ≤ 0 . ( 25 ) 170 Algorithm 1 Learning from complementary labels by exclusion

Summary: This paper deals with the problem of complementary label learning, that is, when we know the set of labels which a given observation does not belong to. In particular, the paper proposes a robust loss function and an algorithm for learning from complimentary labels. Results shown on MNIST and CIFAR datasets indicate the superior accuracy using the proposed loss function.