Rethinking Class-Prior Estimation for Positive-Unlabeled Learning

1 INTRODUCTION . Positive-unlabeled ( PU ) learning can date back to 1990s ( Denis , 1998 ; De Comité et al. , 1999 ; Letouzey et al. , 2000 ) , and there has been a surge of interest in this learning scenario in recent years because of the difficulty to annotate large-scale datasets ( Ren et al. , 2014 ; du Plessis et al. , 2014 ; 2015 ; Christoffel et al. , 2016 ; Jain et al. , 2016 ; Ramaswamy et al. , 2016 ; Sakai et al. , 2018 ; Kato et al. , 2018 ; Bekker & Davis , 2018 ; Gong et al. , 2019 ; Bai et al. , 2021 ; Xia et al. , 2021 ; Yao et al. , 2021 ) . It is also fallen into different applications , such as knowledge-base completion ( Galárraga et al. , 2015 ; Neelakantan et al. , 2015 ) , text classification ( Lee & Liu , 2003 ; Li & Liu , 2003 ) , and medical diagnosis ( Claesen et al. , 2015 ; Zuluaga et al. , 2011 ) . PU learning can be divided into two different settings based on different data generation processes . The first setting is called censoring PU learning ( Elkan & Noto , 2008 ) , which follows a one-sample configuration . Specifically , a sample S is randomly drawn from the unlabeled data distribution Pu , and a positive sample Sp is then distilled from it , i.e. , randomly selecting some positive instances contained in the unlabeled data to be the positive sample . The second setting is called case-control PU learning ( Kiryo et al. , 2017 ) . In this setting , a positive sample Sp = { xi } ki=1 is randomly drawn from the positive class-conditional distribution Pp = P ( X|Y = 1 ) , and an unlabeled sample Su = { xi } ni=k+1 is randomly drawn from the unlabeled data distribution Pu . Because case-control PU learning is more general than censoring PU learning ( Niu et al. , 2016 ) , therefore , we will focus on the setting of case-control PU learning . Under the setting of case-control PU learning , a lot of classification methods have been proposed ( Ren et al. , 2014 ; du Plessis et al. , 2014 ; 2015 ; Christoffel et al. , 2016 ; Sakai et al. , 2018 ; Kato et al. , 2018 ; Bekker & Davis , 2018 ; Kwon et al. , 2019 ; Tanielian & Vasile , 2019 ; Gong et al. , 2019 ) . †Correspondence to Tongliang Liu ( tongliang.liu @ sydney.edu.au ) . However , the class-prior estimation ( CPE ) ( Elkan & Noto , 2008 ; Jain et al. , 2016 ; Ramaswamy et al. , 2016 ; Christoffel et al. , 2016 ; Kato et al. , 2018 ) has received less attention . Formally , CPE is defined as a problem of estimating π = P ( y = 1 ) ∈ ( 0 , 1 ) given a sample from the marginal distribution Pu and a sample from positive class-conditional distribution Pp . The marginal distribution Pu is mixed with both positive and negative class-conditional distributions , i.e. , Pu = πPp+ ( 1−π ) Pn . CPE acts as a crucial building block for state-of-the-art PU classification methods , and it is essential to build statistically-consistent PU classifiers ( du Plessis et al. , 2014 ; Scott , 2015 ; Jain et al. , 2016 ; Kiryo et al. , 2017 ) . The formulation of these classification methods involves the class-prior π , but π is usually unknown in practice . If π is poorly estimated , the classification accuracy of the state-of-the-art PU classification methods ( du Plessis et al. , 2014 ; 2015 ; Kiryo et al. , 2017 ) could be degraded . The mixture proportion estimation ( MPE ) is closely related to CPE ( Blanchard et al. , 2010 ; Scott , 2015 ) . In the setting of MPE , there is a mixture distribution F = ( 1− κ∗ ) G+ κ∗H , ( 1 ) where H and G are called component distributions . Given the samples randomly drawn from F and H , respectively , MPE aims to estimate the maximum proportion κ∗ ∈ ( 0 , 1 ) of H in F . Thereby , if the maximum proportion κ∗ is identical to the class-prior π , the MPE methods can be employed to obtain π by letting Pu and Pp be the mixture distribution F and the component distribution H , respectively ; otherwise , the MPE methods can not be employed . To the best of our knowledge , most of state-of-the-art CPE methods ( Blanchard et al. , 2010 ; Liu & Tao , 2015 ; Scott , 2015 ; Ramaswamy et al. , 2016 ; Jain et al. , 2016 ) are based on MPE , which do not rely on assumptions that the data are drawn from a given parametric family of probability distributions ( i.e. , they are distributional-assumption-free methods ) . To let these distributional-assumption-free methods can be used to identify class-prior π , κ∗ must be identical to the class-prior π . The irreducibility assumption ( Blanchard et al. , 2010 ) has been proposed to make them identical , which is employed by all these CPE methods implicitly or explicitly , to the best of our knowledge . It assumes that the support of the positive class-conditional distribution Pp is not contained in the support of the negative class-conditional distribution Pn . However , it is strong and hard to be verified in PU learning , since Pn is a latent distribution , such that we do not have any prior knowledge about it . Additionally , since the applications of PU learning are diverse ( Hsieh et al. , 2019 ; Bekker & Davis , 2020 ) , it is hard to guarantee that the support of Pp is not in the support of Pn . If the irreducibility assumption can not be satisfied , the existing distributional-assumption-free CPE methods will suffer from an overestimation of π . For example , in Figure 1 ( a ) , we show both the unlabeled data distribution Pu and the component distribution Pp . In Figure 1 ( b ) , we assume the latent negative class-conditional distribution Pn is fixed as shown in the green color , and the positive class-prior π = 0.5 . In Figure 1 ( c ) , we show the existing distributional-assumption-free CPE methods will output the biased class-prior 0.7 . It is different from the ground truth 0.5 , since the support of Pp is contained in the support of Pn . When the irreducibility assumption is not held , how to improve the estimations of distributional-assumption-free PU learning methods is challenging but useful . Because the irreducibility assumption is impossible to check without making any assumption on Pn . Thereby , in this paper , we rethink those CPE methods and propose a novel method called Regrouping CPE ( ReCPE ) which improves the estimations of the current PU learning methods without irreducibility assumption . The main idea of our method is that , instead of estimating the maximum proportion of Pp in Pu , we build a new CPE problem by creating a new auxiliary distribution Pp′ always guaranteeing the irreducibility assumption . Then we use the existing CPE method to obtain the maximum proportion of Pp′ in Pu , which is denoted by π′ . We show that , with both theoretical analyses and experimental validations , when the irreducibility assumption holds , our ReCPE method does not affect the prediction of the existing estimators ; when the irreducibility assumption does not hold , our method will help the current estimators have less estimation bias , which could improve the performances of PU classification tasks . For example , in Figure 1 ( d ) , we create a new class-conditional ( auxiliary ) distribution Pp′ . By solving it , π′ = 0.51 . The estimation bias of the existing estimators will reduce to π′ − π = 0.01 instead of κ∗ − π = 0.2 . The rest of the paper is organized as follows . In Section 2 , we review the irreducibility assumption and its variants . We discuss the difficulty of checking the assumptions . In Section 3 , we provide the estimation biases of the existing consistent distributional-assumption-free CPE methods . Then we propose our method ReCPE , followed by theoretically analysis of its estimation bias and the implementation details . All the proofs are listed in Appendix A . The experimental validations are given in Section 4 . Section 5 concludes the paper . 2 IRREDUCIBILITY OF CPE . In this section , we briefly review the assumptions used for existing distributional-assumption-free CPE estimators . Then we provide the estimation bias introduced by consistent distributional-assumptionfree CPE methods when the assumptions do not hold . The irreducibility assumption . Let Pp and Pu be probability measures ( distributions ) on a measurable space ( X , S ) , where X is the sample space , and S is the σ-algebra . Let κ∗ be the maximum proportion of Pp in Pu . To let κ∗ be identical to π , the irreducibility assumption was proposed by Blanchard et al . ( 2010 ) . Definition 1 ( Irreducibility ) . Pn and Pp are said to satisfy the irreducibility assumption if Pn is not a mixture containing Pp . That is , there does not exist a decomposition Pn = ( 1− β ) Q+ βPp , where Q is a probability distribution on the measurable space ( X , S ) , and 0 < β ≤ 1 . Equivalently , the assumption assumes the support of Pp is hardly contained in the support of Pn . It means that with the selection of different sets S , the probability Pn ( S ) can be arbitrarily close to 0 , and Pp ( S ) > 0 . Suppose we can access the distributions Pu , Pp and the set C containing all possible latent distributions , then the class-prior π can be found as follows : π = κ∗ , sup { α|Pu = ( 1− α ) K + αPp , K ∈ C } = inf S∈S , Pp ( S ) > 0 Pu ( S ) Pp ( S ) . ( 2 ) To the best of our knowledge , all existing distributional-assumption-free CPE methods ( Blanchard et al. , 2010 ; Scott et al. , 2013 ; Liu & Tao , 2015 ; Scott , 2015 ; Ramaswamy et al. , 2016 ; Ivanov , 2019 ) are variants of estimating the maximum proportion κ∗ of Pp in Pu . Many of them are statistically consistent estimators ( Blanchard et al. , 2010 ; Scott et al. , 2013 ; Liu & Tao , 2015 ; Scott , 2015 ) . The variants of the irreducibility . Based on the irreducibility assumption , estimators can be designed with theoretical guarantees that they will converge to the class-prior π ( Blanchard et al. , 2010 ) . However , the convergence rate can be arbitrarily slow ( Scott , 2015 ) . The reason is that the irreducibility assumption implies the following fact ( Blanchard et al. , 2010 ; Scott et al. , 2013 ) inf S∈S , Pp ( S ) > 0 Pn ( S ) Pp ( S ) = 0 , ( 3 ) i.e. , the maximum proportion of Pp in Pn approaches to 0 . To obtain the class-prior π , it requires finding a sequence of the sets S converging to the infimum , which empirically can be hard to find . Therefore , the convergence rate of the designed estimators based on Eq . ( 3 ) will be arbitrarily slow . To ensure a fixed rate of convergence , the anchor set assumption , a stronger variant of the irreducibility assumption , has been proposed ( Scott , 2015 ; Liu & Tao , 2015 ; Xia et al. , 2019 ; 2020 ; Yao et al. , 2020 ) . It assumes that min S∈S , Pp ( S ) > 0 Pn ( S ) Pp ( S ) = 0 , ( 4 ) i.e. , there exists a set can achieve the minimum 0 , which is called an anchor set . Another stronger variant is the separability assumption ( Ramaswamy et al. , 2016 ) which extends the anchor set assumption to a function space . It is proposed to bound the convergence rate of the method based on kernel-mean-matching ( KMM ) technique ( Gretton et al. , 2012 ) .

Summary: This paper addresses the problem of class-prior estimation (CPE) in positive-unlabeled (PU) learning. The authors consider that the existing methods usually fail if the data distribution dissatisfies the irreducibility assumption. To address this problem, the authors introduce a method named Regroping CPE (ReCPE) which tries to transform the original data distribution to an auxiliary data distribution, such that the produced distribution always guaranteeing the irreducibility assumption.   A practice implementation is proposed by firstly training a binary classifier and then picking some samples whose outputs are mostly dissimilar to the negative examples as the pseudo positive samples. The proposed methods can be considered as a pre-processing method and used together with any CPE method. The experiments on synthetic data show the intuition of the proposed ReCPE and the results on real-world data show the effectiveness of the proposed method when it is combined with other CPE methods.