Pointwise Binary Classification with Pairwise Confidence Comparisons

1 INTRODUCTION . Traditional supervised learning techniques have achieved great advances , while they are demanding for precisely labeled data . In many real-world scenarios , it may be too difficult to collect such data . To alleviate this issue , a large number of weakly supervised learning problems ( Zhou , 2018 ) have been extensively studied , including semi-supervised learning ( Zhu & Goldberg , 2009 ; Niu et al. , 2013 ; Sakai et al. , 2018 ) , multi-instance learning ( Zhou et al. , 2009 ; Sun et al. , 2016 ; Zhang & Zhou , 2017 ) , noisy-label learning ( Han et al. , 2018 ; Xia et al. , 2019 ; Wei et al. , 2020 ) , partial-label learning ( Zhang et al. , 2017 ; Feng et al. , 2020b ; Lv et al. , 2020 ) , complementary-label learning ( Ishida et al. , 2017 ; Yu et al. , 2018 ; Ishida et al. , 2019 ; Feng et al. , 2020a ) , positive-unlabeled classification ( Gong et al. , 2019 ) , positive-confidence classification ( Ishida et al. , 2018 ) , similarunlabeled classification ( Bao et al. , 2018 ) , unlabeled-unlabeled classification ( Lu et al. , 2019 ; 2020 ) , and triplet classification ( Cui et al. , 2020 ) . This paper considers another novel weakly supervised learning setting called pairwise comparison ( Pcomp ) classification , where we aim to perform pointwise binary classification with only pairwise comparison data , instead of pointwise labeled data . A pairwise comparison ( x , x′ ) represents that the instance x has a larger confidence of belonging to the positive class than the instance x′ . Such weak supervision ( pairwise confidence comparison ) could be much easier for people to collect than full supervision ( pointwise label ) in practice , especially for applications on sensitive or private matters . For example , it may be difficult to collect sensitive or private data with pointwise labels , as asking for the true labels could be prohibited or illegal . In this case , it could be easier for people to collect other weak supervision like the comparison information between two examples . It is also advantageous to consider pairwise confidence comparisons in pointwise binary classification with class overlapping , where the labeling task becomes difficult , and even experienced labelers may provide wrong pointwise labels . Let us denote the labeling standard of a labeler as p̃ ( y|x ) and assume that an instance x1 is more positive than another instance x2 . Facing the difficult labeling task , different labelers may hold different labeling standards , p̃ ( y = +1|x1 ) > p̃ ( y = +1|x2 ) > 1/2 , p̃ ( y = +1|x1 ) > 1/2 > p̃ ( y = +1|x2 ) , and 1/2 > p̃ ( y = +1|x1 ) > p̃ ( y = +1|x2 ) , thereby providing different pointwise labels : ( +1 , +1 ) , ( +1 , −1 ) , ( −1 , −1 ) . We can find that different labelers may provide inconsistent pointwise labels , while pairwise confidence comparisons are unanimous and accurate . One may argue that we could aggregate multiple labels of the same instance using crowdsourcing learning methods ( Whitehill et al. , 2009 ; Raykar et al. , 2010 ) . However , as not every instance will be labeled by multiple labelers , it is not always applicable to crowdsourcing learning methods . Therefore , our proposed Pcomp classification is useful in this case . Our contributions in this paper can be summarized as follows : • We propose Pcomp classification , a novel weakly supervised learning setting , and present a mathematical formulation for the generation process of pairwise comparison data . • We prove that an unbiased risk estimator ( URE ) can be derived , propose an empirical risk minimization ( ERM ) based method , and present an improvement using correction functions ( Lu et al. , 2020 ) for alleviating overftting when complex models are used . • We start from the noisy-label learning perspective to introduce the RankPruning method ( Northcutt et al. , 2017 ) that holds a progressive URE for solving our proposed Pcomp classification problem and improve it by imposing consistency regularization . • We experimentally demonstrate the effectiveness of our proposed solutions for Pcomp classification . 2 PRELIMINARIES . Binary classification with pairwise comparisons and extra pointwise labels has been studied ( Xu et al. , 2017 ; Kane et al. , 2017 ) . Our paper focuses on a more challenging problem where only pairwise comparison examples are provided . Unlike previous studies ( Xu et al. , 2017 ; Kane et al. , 2017 ) that leverage some pointwise labels to differentiate the labels of pairwise comparisons , our methods are purely based on ERM with only pairwise comparisons . In the next , we briefly introduce some notations and review the related problem formulations of binary classification , positive-unlabeled classification , and unlabeled-unlabeled classification . Binary Classification . Since our paper focuses on how to train a binary classifier from pairwise comparison data , we first review the problem formulation of binary classification . Let the feature space be X and the label space be Y = { +1 , −1 } . Suppose the collected dataset is denoted by D = { ( xi , yi ) } ni=1 where each example ( xi , yi ) is independently sampled from the joint distribution with density p ( x , y ) , which includes an instance xi ∈ X and a label yi ∈ Y . The goal of binary classification is to train an optimal classifier f : X 7→ R by minimizing the following expected classification risk : R ( f ) = Ep ( x , y ) [ ` ( f ( x ) , y ) ] = π+Ep+ ( x ) [ ` ( f ( x ) , +1 ) ] + π−Ep− ( x ) [ ` ( f ( x ) , −1 ) ] , ( 1 ) where ` : R × Y 7→ R+ denotes a binary loss function , π+ : = p ( y = +1 ) ( or π− : = p ( y = −1 ) ) denotes the positive ( or negative ) class prior probability , and p+ ( x ) : = p ( x|y = +1 ) ( or p− ( x ) : = p ( x|y = −1 ) ) denotes the class-conditional probability density of the positive ( or negative ) data . ERM approximates the expectations over p+ ( x ) and p− ( x ) by the empirical averages of positive and negative data and the empirical risk is minimized with respect to the classifier f . Positive-Unlabeled ( PU ) Classification . In some real-world scenarios , it may be difficult to collect negative data , and only positive ( P ) and unlabeled ( U ) data are available . PU classification aims to train an effective binary classifier in this weakly supervised setting . Previous studies ( du Plessis et al. , 2014 ; 2015 ; Kiryo et al. , 2017 ) showed that the classification risk R ( f ) in Eq . ( 1 ) can be rewritten only in terms of positive and unlabeled data as R ( f ) = RPU ( f ) = π+Ep+ ( x ) [ ` ( f ( x ) , +1 ) − ` ( f ( x ) , −1 ) ] + Ep ( x ) [ ` ( f ( x ) , −1 ) ] , ( 2 ) where p ( x ) = π+p+ ( x ) + π−p− ( x ) denotes the probability density of unlabeled data . This risk expression immediately allows us to employ ERM in terms of positive and unlabeled data . Unlabeled-Unlabeled ( UU ) Classification . The recent studies ( Lu et al. , 2019 ; 2020 ) showed that it is possible to train a binary classifier only from two unlabeled datasets with different class priors . Lu et al . ( 2019 ) showed that the classification risk can be rewritten as R ( f ) = RUU ( f ) = Eptr ( x ) [ ( 1− θ′ ) π+ θ − θ′ ` ( f ( x ) , +1 ) − θ ′ ( 1− π+ ) θ − θ′ ` ( f ( x ) , −1 ) ] + Eptr′ ( x′ ) [ θ ( 1− π+ ) θ − θ′ ` ( f ( x′ ) , −1 ) − ( 1− θ ) π+ θ − θ′ ` ( f ( x′ ) , +1 ) ] , ( 3 ) where θ and θ′ are different class priors of two unlabeled datasets , and ptr ( x ) and ptr′ ( x′ ) are the densities of two datasets of unlabeled data , respectively . This risk expression immediately allows us to employ ERM only from two sets of unlabeled data . For RUU ( f ) in Eq . ( 3 ) , if we set θ = 1 , θ′ = π+ , and replace ptr ( x ) and ptr′ ( x′ ) by p+ ( x ) and p ( x ) respectively , then we can recover RPU ( f ) in Eq . ( 2 ) . Therefore , UU classification could be taken as a generalized framework of PU classification in terms of URE . Besides , Eq . ( 3 ) also recovers a complicated URE of similarunlabeled classification ( Bao et al. , 2018 ) by setting θ = π+ and θ′ = π2+/ ( 2π 2 + − 2π+ + 1 ) . To solve our proposed Pcomp classification problem , we will present a mathematical formulation for the generation process of pairwise comparison data , based on which we will explore two UREs to train a binary classifier by ERM and establish the corresponding estimation error bounds . 3 DATA GENERATION PROCESS . In order to derive UREs for performing ERM , we first formulate the underlying generation process of pairwise comparison data1 , which consists of pairs of unlabeled data that we know which one is more likely to be positive . Suppose the provided dataset is denoted by D̃ = { ( xi , x′i ) } ni=1 where ( xi , x′i ) ( with their unknown true labels ( yi , y′i ) ) is expected to satisfy p ( yi = +1|xi ) > p ( y′i = +1|x′i ) . It is clear that we could easily collect pairwise comparison data if the positive confidence ( i.e. , p ( y = +1|x ) ) of each instance could be obtained . However , such information is much harder to obtain than class labels in real-world scenarios . Therefore , unlike some studies ( Ishida et al. , 2018 ; Shinoda et al. , 2020 ) that assume the positive confidence of each instance is provided by the labeler , we only assume that the labeler has access to the labels of training data . Specifically , we adopt the assumption ( Cui et al. , 2020 ) that weakly supervised examples are first sampled from the true data distribution , but the labels are only accessible to the labeler . Then , the labeler would provide us weakly supervised information ( i.e. , pairwise comparison information ) according to the labels of sampled data pairs . That is , for any pair of unlabeled data ( x , x′ ) , the labeler would tell us whether ( x , x′ ) could be collected as a pairwise comparison for Pcomp classification , based on the labels ( y , y′ ) rather than the positive confidences ( p ( y = +1|x ) , p ( y = +1|x′ ) ) . Now , the question becomes : how does the labeler consider ( x , x′ ) as a pairwise comparison for Pcomp classification , in terms of the labels ( y , y′ ) ? As shown in our previous example of binary classification with class overlapping , we could infer that the labels ( y , y′ ) of our required pairwise comparison data ( x , x′ ) for Pcomp classification can only be one of the three cases { ( +1 , −1 ) , ( +1 , +1 ) , ( −1 , −1 ) } , because the condition p ( y = +1|x ) ≥ p ( y′ = +1|x′ ) is definitely violated if ( y , y′ ) = ( −1 , +1 ) . Therefore , we assume that the labeler would take ( x , x′ ) as a pairwise comparison example in the dataset D̃ , if the labels ( y , y′ ) of ( x , x′ ) belong to the above three cases . It is also worth noting that for a pair of data ( x , x′ ) with labels ( y , y′ ) = ( −1 , +1 ) , the labeler would take ( x′ , x ) as a pairwise comparison example . Because by exchanging the positions of ( x , x′ ) , ( x′ , x ) would be associated with labels ( +1 , −1 ) , which belong to the three cases . In summary , we assume that pairwise comparison data are sampled from those pairs of data whose labels belong to the three cases { ( +1 , −1 ) , ( +1 , +1 ) , ( −1 , −1 ) } . Based on the above described generation process of pairwise comparison data , we have the following theorem . Theorem 1 . According to the generation process of pairwise comparison data described above , let p̃ ( x , x′ ) = q ( x , x′ ) π2+ + π 2 − + π+π− , ( 4 ) where q ( x , x′ ) = π2+p+ ( x ) p+ ( x ′ ) + π2−p− ( x ) p− ( x ′ ) + π+π−p+ ( x ) p− ( x ′ ) . Then we have D̃ = { ( xi , x′i ) } ni=1 i.i.d.∼ p̃ ( x , x′ ) . 1In contrast to Xu et al . ( 2019 ) and Xu et al . ( 2020 ) which utilized pairwise comparison data to solve the regression problem , we focus on binary classification . The proof is provided in Appendix A. Theorem 1 provides an explicit expression of the probability density of pairwise comparison data . Next , we would like to extract pointwise information from pairwise information , since our goal is to perform pointwise binary classification . Let π̃ = π2+ +π 2 −+π+π− = π+ +π 2 − = π 2 + +π− and we denote the pointwise data collected from D̃ = { ( xi , x′i ) } ni=1 by breaking the pairwise comparison relation as D̃+ = { xi } ni=1 and D̃− = { x′i } ni=1 . Then we can obtain the following theorem . Theorem 2 . Pointwise examples in D̃+ and D̃− are independently drawn from p̃+ ( x ) and p̃− ( x′ ) , where p̃+ ( x ) = π+ π2− + π+ p+ ( x ) + π2− π2− + π+ p− ( x ) , p̃− ( x ′ ) = π2+ π2+ + π− p+ ( x ′ ) + π− π2+ + π− p− ( x ′ ) . The proof is provided in Appendix B. Theorem 2 shows the relationships between the pointwise densities and the class-conditional densities . Besides , it indicates that from pairwise comparison data , we can essentially obtain examples that are independently drawn from p̃+ ( x ) and p̃− ( x′ ) .

Summary: The paper develops a method to learn a binary classifier based only on pairwise comparison data. For example, the classifier learns to classify pictures of people as "adult" versus "child" based on pairwise comparisons of the form "person C is older than person X". The authors derive their method based on an empirical risk minimization argument. The authors test their methods on four standard data sets (three MNIST variants and one more). They compare to some baselines including binary biased, noisy unbiased, and RankPruning. They try 4 different variations of their method. The Pcomp-Teacher model performs especially well.