Rethinking Class-Prior Estimation for Positive-Unlabeled Learning

1 INTRODUCTION . Positive-unlabeled ( PU ) learning can date back to 1990s ( Denis , 1998 ; De Comité et al. , 1999 ; Letouzey et al. , 2000 ) , and there has been a surge of interest in this learning scenario in recent years because of the difficulty to annotate large-scale datasets ( Ren et al. , 2014 ; du Plessis et al. , 2014 ; 2015 ; Christoffel et al. , 2016 ; Jain et al. , 2016 ; Ramaswamy et al. , 2016 ; Sakai et al. , 2018 ; Kato et al. , 2018 ; Bekker & Davis , 2018 ; Gong et al. , 2019 ; Bai et al. , 2021 ; Xia et al. , 2021 ; Yao et al. , 2021 ) . It is also fallen into different applications , such as knowledge-base completion ( Galárraga et al. , 2015 ; Neelakantan et al. , 2015 ) , text classification ( Lee & Liu , 2003 ; Li & Liu , 2003 ) , and medical diagnosis ( Claesen et al. , 2015 ; Zuluaga et al. , 2011 ) . PU learning can be divided into two different settings based on different data generation processes . The first setting is called censoring PU learning ( Elkan & Noto , 2008 ) , which follows a one-sample configuration . Specifically , a sample S is randomly drawn from the unlabeled data distribution Pu , and a positive sample Sp is then distilled from it , i.e. , randomly selecting some positive instances contained in the unlabeled data to be the positive sample . The second setting is called case-control PU learning ( Kiryo et al. , 2017 ) . In this setting , a positive sample Sp = { xi } ki=1 is randomly drawn from the positive class-conditional distribution Pp = P ( X|Y = 1 ) , and an unlabeled sample Su = { xi } ni=k+1 is randomly drawn from the unlabeled data distribution Pu . Because case-control PU learning is more general than censoring PU learning ( Niu et al. , 2016 ) , therefore , we will focus on the setting of case-control PU learning . Under the setting of case-control PU learning , a lot of classification methods have been proposed ( Ren et al. , 2014 ; du Plessis et al. , 2014 ; 2015 ; Christoffel et al. , 2016 ; Sakai et al. , 2018 ; Kato et al. , 2018 ; Bekker & Davis , 2018 ; Kwon et al. , 2019 ; Tanielian & Vasile , 2019 ; Gong et al. , 2019 ) . †Correspondence to Tongliang Liu ( tongliang.liu @ sydney.edu.au ) . However , the class-prior estimation ( CPE ) ( Elkan & Noto , 2008 ; Jain et al. , 2016 ; Ramaswamy et al. , 2016 ; Christoffel et al. , 2016 ; Kato et al. , 2018 ) has received less attention . Formally , CPE is defined as a problem of estimating π = P ( y = 1 ) ∈ ( 0 , 1 ) given a sample from the marginal distribution Pu and a sample from positive class-conditional distribution Pp . The marginal distribution Pu is mixed with both positive and negative class-conditional distributions , i.e. , Pu = πPp+ ( 1−π ) Pn . CPE acts as a crucial building block for state-of-the-art PU classification methods , and it is essential to build statistically-consistent PU classifiers ( du Plessis et al. , 2014 ; Scott , 2015 ; Jain et al. , 2016 ; Kiryo et al. , 2017 ) . The formulation of these classification methods involves the class-prior π , but π is usually unknown in practice . If π is poorly estimated , the classification accuracy of the state-of-the-art PU classification methods ( du Plessis et al. , 2014 ; 2015 ; Kiryo et al. , 2017 ) could be degraded . The mixture proportion estimation ( MPE ) is closely related to CPE ( Blanchard et al. , 2010 ; Scott , 2015 ) . In the setting of MPE , there is a mixture distribution F = ( 1− κ∗ ) G+ κ∗H , ( 1 ) where H and G are called component distributions . Given the samples randomly drawn from F and H , respectively , MPE aims to estimate the maximum proportion κ∗ ∈ ( 0 , 1 ) of H in F . Thereby , if the maximum proportion κ∗ is identical to the class-prior π , the MPE methods can be employed to obtain π by letting Pu and Pp be the mixture distribution F and the component distribution H , respectively ; otherwise , the MPE methods can not be employed . To the best of our knowledge , most of state-of-the-art CPE methods ( Blanchard et al. , 2010 ; Liu & Tao , 2015 ; Scott , 2015 ; Ramaswamy et al. , 2016 ; Jain et al. , 2016 ) are based on MPE , which do not rely on assumptions that the data are drawn from a given parametric family of probability distributions ( i.e. , they are distributional-assumption-free methods ) . To let these distributional-assumption-free methods can be used to identify class-prior π , κ∗ must be identical to the class-prior π . The irreducibility assumption ( Blanchard et al. , 2010 ) has been proposed to make them identical , which is employed by all these CPE methods implicitly or explicitly , to the best of our knowledge . It assumes that the support of the positive class-conditional distribution Pp is not contained in the support of the negative class-conditional distribution Pn . However , it is strong and hard to be verified in PU learning , since Pn is a latent distribution , such that we do not have any prior knowledge about it . Additionally , since the applications of PU learning are diverse ( Hsieh et al. , 2019 ; Bekker & Davis , 2020 ) , it is hard to guarantee that the support of Pp is not in the support of Pn . If the irreducibility assumption can not be satisfied , the existing distributional-assumption-free CPE methods will suffer from an overestimation of π . For example , in Figure 1 ( a ) , we show both the unlabeled data distribution Pu and the component distribution Pp . In Figure 1 ( b ) , we assume the latent negative class-conditional distribution Pn is fixed as shown in the green color , and the positive class-prior π = 0.5 . In Figure 1 ( c ) , we show the existing distributional-assumption-free CPE methods will output the biased class-prior 0.7 . It is different from the ground truth 0.5 , since the support of Pp is contained in the support of Pn . When the irreducibility assumption is not held , how to improve the estimations of distributional-assumption-free PU learning methods is challenging but useful . Because the irreducibility assumption is impossible to check without making any assumption on Pn . Thereby , in this paper , we rethink those CPE methods and propose a novel method called Regrouping CPE ( ReCPE ) which improves the estimations of the current PU learning methods without irreducibility assumption . The main idea of our method is that , instead of estimating the maximum proportion of Pp in Pu , we build a new CPE problem by creating a new auxiliary distribution Pp′ always guaranteeing the irreducibility assumption . Then we use the existing CPE method to obtain the maximum proportion of Pp′ in Pu , which is denoted by π′ . We show that , with both theoretical analyses and experimental validations , when the irreducibility assumption holds , our ReCPE method does not affect the prediction of the existing estimators ; when the irreducibility assumption does not hold , our method will help the current estimators have less estimation bias , which could improve the performances of PU classification tasks . For example , in Figure 1 ( d ) , we create a new class-conditional ( auxiliary ) distribution Pp′ . By solving it , π′ = 0.51 . The estimation bias of the existing estimators will reduce to π′ − π = 0.01 instead of κ∗ − π = 0.2 . The rest of the paper is organized as follows . In Section 2 , we review the irreducibility assumption and its variants . We discuss the difficulty of checking the assumptions . In Section 3 , we provide the estimation biases of the existing consistent distributional-assumption-free CPE methods . Then we propose our method ReCPE , followed by theoretically analysis of its estimation bias and the implementation details . All the proofs are listed in Appendix A . The experimental validations are given in Section 4 . Section 5 concludes the paper . 2 IRREDUCIBILITY OF CPE . In this section , we briefly review the assumptions used for existing distributional-assumption-free CPE estimators . Then we provide the estimation bias introduced by consistent distributional-assumptionfree CPE methods when the assumptions do not hold . The irreducibility assumption . Let Pp and Pu be probability measures ( distributions ) on a measurable space ( X , S ) , where X is the sample space , and S is the σ-algebra . Let κ∗ be the maximum proportion of Pp in Pu . To let κ∗ be identical to π , the irreducibility assumption was proposed by Blanchard et al . ( 2010 ) . Definition 1 ( Irreducibility ) . Pn and Pp are said to satisfy the irreducibility assumption if Pn is not a mixture containing Pp . That is , there does not exist a decomposition Pn = ( 1− β ) Q+ βPp , where Q is a probability distribution on the measurable space ( X , S ) , and 0 < β ≤ 1 . Equivalently , the assumption assumes the support of Pp is hardly contained in the support of Pn . It means that with the selection of different sets S , the probability Pn ( S ) can be arbitrarily close to 0 , and Pp ( S ) > 0 . Suppose we can access the distributions Pu , Pp and the set C containing all possible latent distributions , then the class-prior π can be found as follows : π = κ∗ , sup { α|Pu = ( 1− α ) K + αPp , K ∈ C } = inf S∈S , Pp ( S ) > 0 Pu ( S ) Pp ( S ) . ( 2 ) To the best of our knowledge , all existing distributional-assumption-free CPE methods ( Blanchard et al. , 2010 ; Scott et al. , 2013 ; Liu & Tao , 2015 ; Scott , 2015 ; Ramaswamy et al. , 2016 ; Ivanov , 2019 ) are variants of estimating the maximum proportion κ∗ of Pp in Pu . Many of them are statistically consistent estimators ( Blanchard et al. , 2010 ; Scott et al. , 2013 ; Liu & Tao , 2015 ; Scott , 2015 ) . The variants of the irreducibility . Based on the irreducibility assumption , estimators can be designed with theoretical guarantees that they will converge to the class-prior π ( Blanchard et al. , 2010 ) . However , the convergence rate can be arbitrarily slow ( Scott , 2015 ) . The reason is that the irreducibility assumption implies the following fact ( Blanchard et al. , 2010 ; Scott et al. , 2013 ) inf S∈S , Pp ( S ) > 0 Pn ( S ) Pp ( S ) = 0 , ( 3 ) i.e. , the maximum proportion of Pp in Pn approaches to 0 . To obtain the class-prior π , it requires finding a sequence of the sets S converging to the infimum , which empirically can be hard to find . Therefore , the convergence rate of the designed estimators based on Eq . ( 3 ) will be arbitrarily slow . To ensure a fixed rate of convergence , the anchor set assumption , a stronger variant of the irreducibility assumption , has been proposed ( Scott , 2015 ; Liu & Tao , 2015 ; Xia et al. , 2019 ; 2020 ; Yao et al. , 2020 ) . It assumes that min S∈S , Pp ( S ) > 0 Pn ( S ) Pp ( S ) = 0 , ( 4 ) i.e. , there exists a set can achieve the minimum 0 , which is called an anchor set . Another stronger variant is the separability assumption ( Ramaswamy et al. , 2016 ) which extends the anchor set assumption to a function space . It is proposed to bound the convergence rate of the method based on kernel-mean-matching ( KMM ) technique ( Gretton et al. , 2012 ) .

Summary: This paper studies a relatively little-concerned problem, that is, the problem of class-prior estimation(CPE) in PU scenes. The authors concluded that the existing CPE methods are based on a critical assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution. However, in actual scenarios, this assumption may not be satisfied. It is also difficult to prove that a certain data set satisfies this assumption. Existing CPE methods will systematically overestimate the class prior when the data does not meet the critical assumption. To remove the assumption to make CPE always valid, a strategy called Regrouping CPE (ReCPE), which builds an auxiliary probability distribution, are proposed so that the support of the positive data distribution is never contained in the support of the negative data distribution. Theoretically, this method can give a more accurate estimate regardless of whether the dataset meets the assumption. The authors also proved the effectiveness of the method through a series of experiments.