Mixture Proportion Estimation and PU Learning:A Modern Approach

1 Introduction . When deploying k-way classifiers in the wild , what can we do when confronted with data from a previously unseen class ( k ` 1 ) ? Theory dictates that learning under distribution shift is impossible absent assumptions . And yet people appear to exhibit this capability routinely . Faced with new surprising symptoms , doctors can recognize the presence of a previously unseen ailment and attempt to estimate its prevalence . Similarly , naturalists can discover new species , estimate their range and population , and recognize them reliably going forward . To begin making this problem tractable , we might make the label shift assumption [ 37 , 41 , 29 ] , i.e. , that while the class balance ppyq can change , the class conditional distributions ppx|yq do not . Moreover , we might begin by focusing on the base case , where only one class has been seen previously , i.e. , k “ 1 . Here , we possess ( labeled ) positive data from the source distribution , and ( unlabeled ) data from the target distribution , consisting of both positive and negative instances . This problem has been studied in the literature as learning from positive and unlabeled data [ 8 , 27 ] and has typically been broken down into two subtasks : ( i ) Mixture Proportion Estimation ( MPE ) where we estimate α , the fraction of positives among the unlabeled examples ; and ( ii ) PU-learning where this estimate is incorporated into a scheme for learning a Positive-versus-Negative ( PvN ) binary classifier . Traditionally , MPE and PU-learning have been motivated by settings involving large databases where unlabeled examples are abundant and a small fraction of the total positives have been extracted . For example , medical records might be annotated indicating the presence of certain diagnoses , but the unmarked passages are not necessarily negative . This setup has also been motivated by protein and 1Code is available at https : //github.com/acmi-lab/PU_learning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . gene identification [ 16 ] . Databases in molecular biology often contain lists of molecules known to exhibit some characteristic of interest . However , many other molecules may exhibit the desired characteristic , even if this remains unknown to science . Many methods have been proposed for both MPE [ 16 , 12 , 39 , 35 , 21 , 4 , 36 , 20 ] and PU-learning [ 14 , 11 , 23 ] . However , classical MPE methods break down in high-dimensional settings [ 35 ] or yield estimators whose accuracy depends on restrictive conditions [ 12 , 39 ] . On the other hand , most recent proposals either lack theoretical coherence , rely on heroic assumptions , or depend precariously on tuning hyperparameters that are , by the very problem setting , untunable . For PU learning , Elkan and Noto [ 16 ] suggest training a classifier to distinguish positive from unlabeled data followed by a rescaling procedure . Du Plessis et al . [ 11 ] suggest an unbiased risk estimation framework for PU learning . However , these methods fail badly when applied with model classes capable of overfitting and thus implementations on high-dimensional datasets rely on extensive hyperparameter tuning and additional ad-hoc heuristics that do not transport effectively across datasets . In this paper , we propose ( i ) Best Bin Estimation ( BBE ) , an effective technique for MPE that produces consistent estimates pα under mild assumptions and admits finite-sample statistical guarantees achieving the desired Op1 { ? nq rates ; and ( ii ) learning with the Conditional Value Ignoring Risk ( CVIR ) objective , which discards the highest loss pα fraction of examples on each training epoch , removing the incentive to overfit to the unlabeled positive examples . Both methods are simple to implement , compatible with arbitrary hypothesis classes ( including deep networks ) , and dominate existing methods in our experimental evaluation . Finally , we combine the two in an iterated Transform-Estimate-Discard ( TED ) n framework that significantly improves both MPE estimation error and classifier error . We build on label shift methods [ 29 , 3 , 2 , 34 , 17 ] , that leverage black-box classifiers to reduce dimensionality , estimating the target label distribution as a functional of source and target pushforward distributions . While label shift methods rely on classifiers trained to separate previously seen classes , BBE is able to exploit a Positive-versus-Unlabeled ( PvU ) target classifier , which gives each input a score indicating how likely it is to be a positive sample . In particular , BBE identifies a threshold such that by estimating the ratio between the fractions of positive and unlabeled points receiving scores above the threshold , we obtain the mixture proportion α. BBE works because in practice , for many datasets , PvU classifiers , even when uncalibrated , produce outputs with near monotonic calibration diagrams . Higher scores correspond to a higher proportion of positives , and when the positive data contains a separable sub-domain , i.e. , a region of the input space where only the positive distribution has support , classifiers often exhibit a threshold above which the top bin contains mostly positive examples . We show that the existence of a ( nearly ) pure top bin is sufficient for BBE to produce a ( nearly ) consistent estimate pα , whose finite sample convergence rates depend on the fraction of examples in the bin and whose bias depends on the purity of the bin . Crucially , we can estimate the optimal threshold from data . We conduct a battery of experiments both to empirically validate our claim that BBE ’ s assumptions are mild and frequently hold in practice , and to establish the outperformance of BBE , CVIR , and ( TED ) n over the previous state of the art . We first motivate BBE by demonstrating that in practice PvU classifiers tend to isolate a reasonably large , reasonably pure top bin . We then conduct extensive experiments on semi-synthetic data , adapting a variety of binary classification datasets to the PU learning setup and demonstrating the superior performance of BBE and PU-learning with the CVIR objective . Moreover , we show that ( TED ) n , which combines the two in an iterative fashion , improves significantly over previous methods across several architectures on a range of image and text datasets . 2 Related Work . Research on MPE and PU learning date to [ 9 , 8 , 27 ] ( see review by [ 5 ] ) . Elkan and Noto [ 16 ] first proposed to leverage a PvU classifier to estimate the mixture proportion . Du Plessis and Sugiyama [ 13 ] propose a different method for estimating the mixture coefficient based on Pearson divergence minimization . While they do not require a PvU classifier , they suffer the same shortcoming . Both methods require that the positive and negative examples have disjoint support . Our requirements are considerably milder . Blanchard et al . [ 6 ] observe that without assumptions on the underlying positive and negative distributions , the mixture proportion is not identifiable . Furthermore , [ 6 ] provide an irreducibility condition that identifies α and propose an estimator that converges to the true α . While their estimator can converge arbitrarily slowly , Scott [ 39 ] showed faster convergence ( Op1 { ? nq ) under stronger conditions . Unfortunately , despite its appealing theoretical properties Blanchard et al . [ 6 ] ’ s estimator is computationally infeasible . Building on Blanchard et al . [ 6 ] , Sanderson and Scott [ 38 ] and Scott [ 39 ] proposed estimating the mixture proportion from a ROC curve constructed for the PvU classifier . However , when the PvU classifier is not perfect , these methods are not clearly understood . Ramaswamy et al . [ 35 ] proposed the first computationally feasible algorithm for MPE with convergence guarantees to the true proportion . Their method KM , requires embedding distributions onto an RKHS . However , their estimator underperforms on high dimensional datasets and scales poorly with large datasets . Bekker and Davis [ 4 ] proposed TIcE , hoping to identify a positive subdomain in the input space using decision tree induction . This method also underperforms in high-dimensional settings . In the most similar works , Jain et al . [ 21 ] and Ivanov [ 20 ] explore dimensionality reduction using a PvU classifier . Both methods estimate α through a procedure operating on the PvU classifier ’ s output . However , neither methods has provided theoretical backing . [ 20 ] concede that their method often fails and returns a zero estimate , requiring that they fall back to a different estimator . Moreover while both papers state that their method require the Bayes-optimal PvU classifier to identify α in the transformed space , we prove that even when hypothesis class is well specified for PvN learning , PvU training can fail to recover the Bayes-optimal scoring function . Furthermore , we also show that the heuristic estimator in Scott [ 39 ] can be thought of as using PvU classifier for dimensionality reduction . While this heuristic is similar to our estimator in spirit , we show that the functional form of their estimator is different from ours and note that their heuristic enjoys no theoretical guarantee . By contrast , our estimator BBE is theoretically coherent under mild conditions and outperforms all of these methods empirically . Given α , Elkan and Noto [ 16 ] propose a transformation via Bayes rule to obtain the PvN classifier . They also propose a weighted objective , with weights given by the PvU classifier . Other propose unbiased risk estimators [ 14 , 11 ] which require the mixture proportion α . Du Plessis et al . [ 14 ] proposed an unbiased estimator with non-convex loss functions satisfying a specific symmetric condition , and subsequently Du Plessis et al . [ 11 ] generalized it to convex loss functions ( denoted uPU in our experiments ) . in our experiments . Noting the problem of overfitting in modern overparameterized models , Kiryo et al . [ 23 ] propose a regularized extension that clips the loss on unlabeled data to zero . This is considered the current state-of-the-art in PU literature ( denoted nnPU in our experiments ) . More recently , Ivanov [ 20 ] proposed DEDPUL , which finetunes the PvU classifiers using several heuristics , Bayes rule , and Expectation Maximization ( EM ) . Since their method only applies a post-processing procedure , they rely on a good domain discriminator classifier in the first place and several hyperparameters for their heuristics . Several classical methods attempt to learn weights that identify reliable negative examples [ 30 , 28 , 26 , 31 , 44 ] . However , these earlier methods have not been successful with modern deep learning models . Algorithm 1 Best Bin Estimation ( BBE ) input : Validation positive ( Xp ) and unlabeled ( Xu ) samples . Blackbox model classifier pf : X Ñ r0 , 1s . Hyperparameter 0 ă δ , γ ă 1 . 1 : Zp , Zu “ fpXpq , fpXuq . 2 : pqupzq , pqppzq “ ř ziPZp Irziězs np , ř ziPZu Irziězs nu for all z P r0 , 1s . 3 : Estimate pc : “ argmincPr0,1s ˆ pqupcq pqppcq ` 1 ` γ pqppcq ˆ b logp4 { δq 2nu ` b logp4 { δq 2np ˙˙ . output : pα : “ pquppcq pqpppcq

Summary: The paper proposes two methods for PU learning. First, a method for estimating the fraction of positives among unlabeled examples, named Best Bin Estimation (BBE). Secondly, an approach for PU learning called Conditional Value Under Optimism (CVuO). The BBE works on top of the decision score of a given pretrained classifier. The score is used to find a region of the instance space with the minimal ratio of the portion of unlabeled and the portion of positive examples. The ratio is an upper bound on the fraction of positives being estimated. In case there exists a region populated only by positive examples (the condition called pure positive bin property), the bound is tight. The authors provide a finite sample bound for their estimator. The CVuO turns any supervised method to PU learning algorithm. The approach assumes the fraction of positive examples to be known. The idea is to rank the unlabeled examples by a score of a pretrained classifier and remove it the fraction of positive examples. The remaining examples are taken as true negatives and then used with the true positives to re-train the classifier. The procedure is repeated until convergence. Besides, the authors propose an algorithm, called $TED^n$, combining BBE and CVuO. The methods are empirically shown to outperform several recent methods on semisynthetic problems created from CIFAR, MNIST, and IMDB datasets.