Mixture Proportion Estimation and PU Learning:A Modern Approach

1 Introduction . When deploying k-way classifiers in the wild , what can we do when confronted with data from a previously unseen class ( k ` 1 ) ? Theory dictates that learning under distribution shift is impossible absent assumptions . And yet people appear to exhibit this capability routinely . Faced with new surprising symptoms , doctors can recognize the presence of a previously unseen ailment and attempt to estimate its prevalence . Similarly , naturalists can discover new species , estimate their range and population , and recognize them reliably going forward . To begin making this problem tractable , we might make the label shift assumption [ 37 , 41 , 29 ] , i.e. , that while the class balance ppyq can change , the class conditional distributions ppx|yq do not . Moreover , we might begin by focusing on the base case , where only one class has been seen previously , i.e. , k “ 1 . Here , we possess ( labeled ) positive data from the source distribution , and ( unlabeled ) data from the target distribution , consisting of both positive and negative instances . This problem has been studied in the literature as learning from positive and unlabeled data [ 8 , 27 ] and has typically been broken down into two subtasks : ( i ) Mixture Proportion Estimation ( MPE ) where we estimate α , the fraction of positives among the unlabeled examples ; and ( ii ) PU-learning where this estimate is incorporated into a scheme for learning a Positive-versus-Negative ( PvN ) binary classifier . Traditionally , MPE and PU-learning have been motivated by settings involving large databases where unlabeled examples are abundant and a small fraction of the total positives have been extracted . For example , medical records might be annotated indicating the presence of certain diagnoses , but the unmarked passages are not necessarily negative . This setup has also been motivated by protein and 1Code is available at https : //github.com/acmi-lab/PU_learning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . gene identification [ 16 ] . Databases in molecular biology often contain lists of molecules known to exhibit some characteristic of interest . However , many other molecules may exhibit the desired characteristic , even if this remains unknown to science . Many methods have been proposed for both MPE [ 16 , 12 , 39 , 35 , 21 , 4 , 36 , 20 ] and PU-learning [ 14 , 11 , 23 ] . However , classical MPE methods break down in high-dimensional settings [ 35 ] or yield estimators whose accuracy depends on restrictive conditions [ 12 , 39 ] . On the other hand , most recent proposals either lack theoretical coherence , rely on heroic assumptions , or depend precariously on tuning hyperparameters that are , by the very problem setting , untunable . For PU learning , Elkan and Noto [ 16 ] suggest training a classifier to distinguish positive from unlabeled data followed by a rescaling procedure . Du Plessis et al . [ 11 ] suggest an unbiased risk estimation framework for PU learning . However , these methods fail badly when applied with model classes capable of overfitting and thus implementations on high-dimensional datasets rely on extensive hyperparameter tuning and additional ad-hoc heuristics that do not transport effectively across datasets . In this paper , we propose ( i ) Best Bin Estimation ( BBE ) , an effective technique for MPE that produces consistent estimates pα under mild assumptions and admits finite-sample statistical guarantees achieving the desired Op1 { ? nq rates ; and ( ii ) learning with the Conditional Value Ignoring Risk ( CVIR ) objective , which discards the highest loss pα fraction of examples on each training epoch , removing the incentive to overfit to the unlabeled positive examples . Both methods are simple to implement , compatible with arbitrary hypothesis classes ( including deep networks ) , and dominate existing methods in our experimental evaluation . Finally , we combine the two in an iterated Transform-Estimate-Discard ( TED ) n framework that significantly improves both MPE estimation error and classifier error . We build on label shift methods [ 29 , 3 , 2 , 34 , 17 ] , that leverage black-box classifiers to reduce dimensionality , estimating the target label distribution as a functional of source and target pushforward distributions . While label shift methods rely on classifiers trained to separate previously seen classes , BBE is able to exploit a Positive-versus-Unlabeled ( PvU ) target classifier , which gives each input a score indicating how likely it is to be a positive sample . In particular , BBE identifies a threshold such that by estimating the ratio between the fractions of positive and unlabeled points receiving scores above the threshold , we obtain the mixture proportion α. BBE works because in practice , for many datasets , PvU classifiers , even when uncalibrated , produce outputs with near monotonic calibration diagrams . Higher scores correspond to a higher proportion of positives , and when the positive data contains a separable sub-domain , i.e. , a region of the input space where only the positive distribution has support , classifiers often exhibit a threshold above which the top bin contains mostly positive examples . We show that the existence of a ( nearly ) pure top bin is sufficient for BBE to produce a ( nearly ) consistent estimate pα , whose finite sample convergence rates depend on the fraction of examples in the bin and whose bias depends on the purity of the bin . Crucially , we can estimate the optimal threshold from data . We conduct a battery of experiments both to empirically validate our claim that BBE ’ s assumptions are mild and frequently hold in practice , and to establish the outperformance of BBE , CVIR , and ( TED ) n over the previous state of the art . We first motivate BBE by demonstrating that in practice PvU classifiers tend to isolate a reasonably large , reasonably pure top bin . We then conduct extensive experiments on semi-synthetic data , adapting a variety of binary classification datasets to the PU learning setup and demonstrating the superior performance of BBE and PU-learning with the CVIR objective . Moreover , we show that ( TED ) n , which combines the two in an iterative fashion , improves significantly over previous methods across several architectures on a range of image and text datasets . 2 Related Work . Research on MPE and PU learning date to [ 9 , 8 , 27 ] ( see review by [ 5 ] ) . Elkan and Noto [ 16 ] first proposed to leverage a PvU classifier to estimate the mixture proportion . Du Plessis and Sugiyama [ 13 ] propose a different method for estimating the mixture coefficient based on Pearson divergence minimization . While they do not require a PvU classifier , they suffer the same shortcoming . Both methods require that the positive and negative examples have disjoint support . Our requirements are considerably milder . Blanchard et al . [ 6 ] observe that without assumptions on the underlying positive and negative distributions , the mixture proportion is not identifiable . Furthermore , [ 6 ] provide an irreducibility condition that identifies α and propose an estimator that converges to the true α . While their estimator can converge arbitrarily slowly , Scott [ 39 ] showed faster convergence ( Op1 { ? nq ) under stronger conditions . Unfortunately , despite its appealing theoretical properties Blanchard et al . [ 6 ] ’ s estimator is computationally infeasible . Building on Blanchard et al . [ 6 ] , Sanderson and Scott [ 38 ] and Scott [ 39 ] proposed estimating the mixture proportion from a ROC curve constructed for the PvU classifier . However , when the PvU classifier is not perfect , these methods are not clearly understood . Ramaswamy et al . [ 35 ] proposed the first computationally feasible algorithm for MPE with convergence guarantees to the true proportion . Their method KM , requires embedding distributions onto an RKHS . However , their estimator underperforms on high dimensional datasets and scales poorly with large datasets . Bekker and Davis [ 4 ] proposed TIcE , hoping to identify a positive subdomain in the input space using decision tree induction . This method also underperforms in high-dimensional settings . In the most similar works , Jain et al . [ 21 ] and Ivanov [ 20 ] explore dimensionality reduction using a PvU classifier . Both methods estimate α through a procedure operating on the PvU classifier ’ s output . However , neither methods has provided theoretical backing . [ 20 ] concede that their method often fails and returns a zero estimate , requiring that they fall back to a different estimator . Moreover while both papers state that their method require the Bayes-optimal PvU classifier to identify α in the transformed space , we prove that even when hypothesis class is well specified for PvN learning , PvU training can fail to recover the Bayes-optimal scoring function . Furthermore , we also show that the heuristic estimator in Scott [ 39 ] can be thought of as using PvU classifier for dimensionality reduction . While this heuristic is similar to our estimator in spirit , we show that the functional form of their estimator is different from ours and note that their heuristic enjoys no theoretical guarantee . By contrast , our estimator BBE is theoretically coherent under mild conditions and outperforms all of these methods empirically . Given α , Elkan and Noto [ 16 ] propose a transformation via Bayes rule to obtain the PvN classifier . They also propose a weighted objective , with weights given by the PvU classifier . Other propose unbiased risk estimators [ 14 , 11 ] which require the mixture proportion α . Du Plessis et al . [ 14 ] proposed an unbiased estimator with non-convex loss functions satisfying a specific symmetric condition , and subsequently Du Plessis et al . [ 11 ] generalized it to convex loss functions ( denoted uPU in our experiments ) . in our experiments . Noting the problem of overfitting in modern overparameterized models , Kiryo et al . [ 23 ] propose a regularized extension that clips the loss on unlabeled data to zero . This is considered the current state-of-the-art in PU literature ( denoted nnPU in our experiments ) . More recently , Ivanov [ 20 ] proposed DEDPUL , which finetunes the PvU classifiers using several heuristics , Bayes rule , and Expectation Maximization ( EM ) . Since their method only applies a post-processing procedure , they rely on a good domain discriminator classifier in the first place and several hyperparameters for their heuristics . Several classical methods attempt to learn weights that identify reliable negative examples [ 30 , 28 , 26 , 31 , 44 ] . However , these earlier methods have not been successful with modern deep learning models . Algorithm 1 Best Bin Estimation ( BBE ) input : Validation positive ( Xp ) and unlabeled ( Xu ) samples . Blackbox model classifier pf : X Ñ r0 , 1s . Hyperparameter 0 ă δ , γ ă 1 . 1 : Zp , Zu “ fpXpq , fpXuq . 2 : pqupzq , pqppzq “ ř ziPZp Irziězs np , ř ziPZu Irziězs nu for all z P r0 , 1s . 3 : Estimate pc : “ argmincPr0,1s ˆ pqupcq pqppcq ` 1 ` γ pqppcq ˆ b logp4 { δq 2nu ` b logp4 { δq 2np ˙˙ . output : pα : “ pquppcq pqpppcq

Summary: This paper targets the problems of mixture proportion estimation and PU learning. The authors propose two simple but effective methods to address two problems. Also, the combination of two methods is proposed to train an accurate binary classifier under mild conditions. A series of experiments are conducted to verify the effectiveness of the proposed method.