Mixture Proportion Estimation and PU Learning:A Modern Approach

1 Introduction . When deploying k-way classifiers in the wild , what can we do when confronted with data from a previously unseen class ( k ` 1 ) ? Theory dictates that learning under distribution shift is impossible absent assumptions . And yet people appear to exhibit this capability routinely . Faced with new surprising symptoms , doctors can recognize the presence of a previously unseen ailment and attempt to estimate its prevalence . Similarly , naturalists can discover new species , estimate their range and population , and recognize them reliably going forward . To begin making this problem tractable , we might make the label shift assumption [ 37 , 41 , 29 ] , i.e. , that while the class balance ppyq can change , the class conditional distributions ppx|yq do not . Moreover , we might begin by focusing on the base case , where only one class has been seen previously , i.e. , k “ 1 . Here , we possess ( labeled ) positive data from the source distribution , and ( unlabeled ) data from the target distribution , consisting of both positive and negative instances . This problem has been studied in the literature as learning from positive and unlabeled data [ 8 , 27 ] and has typically been broken down into two subtasks : ( i ) Mixture Proportion Estimation ( MPE ) where we estimate α , the fraction of positives among the unlabeled examples ; and ( ii ) PU-learning where this estimate is incorporated into a scheme for learning a Positive-versus-Negative ( PvN ) binary classifier . Traditionally , MPE and PU-learning have been motivated by settings involving large databases where unlabeled examples are abundant and a small fraction of the total positives have been extracted . For example , medical records might be annotated indicating the presence of certain diagnoses , but the unmarked passages are not necessarily negative . This setup has also been motivated by protein and 1Code is available at https : //github.com/acmi-lab/PU_learning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . gene identification [ 16 ] . Databases in molecular biology often contain lists of molecules known to exhibit some characteristic of interest . However , many other molecules may exhibit the desired characteristic , even if this remains unknown to science . Many methods have been proposed for both MPE [ 16 , 12 , 39 , 35 , 21 , 4 , 36 , 20 ] and PU-learning [ 14 , 11 , 23 ] . However , classical MPE methods break down in high-dimensional settings [ 35 ] or yield estimators whose accuracy depends on restrictive conditions [ 12 , 39 ] . On the other hand , most recent proposals either lack theoretical coherence , rely on heroic assumptions , or depend precariously on tuning hyperparameters that are , by the very problem setting , untunable . For PU learning , Elkan and Noto [ 16 ] suggest training a classifier to distinguish positive from unlabeled data followed by a rescaling procedure . Du Plessis et al . [ 11 ] suggest an unbiased risk estimation framework for PU learning . However , these methods fail badly when applied with model classes capable of overfitting and thus implementations on high-dimensional datasets rely on extensive hyperparameter tuning and additional ad-hoc heuristics that do not transport effectively across datasets . In this paper , we propose ( i ) Best Bin Estimation ( BBE ) , an effective technique for MPE that produces consistent estimates pα under mild assumptions and admits finite-sample statistical guarantees achieving the desired Op1 { ? nq rates ; and ( ii ) learning with the Conditional Value Ignoring Risk ( CVIR ) objective , which discards the highest loss pα fraction of examples on each training epoch , removing the incentive to overfit to the unlabeled positive examples . Both methods are simple to implement , compatible with arbitrary hypothesis classes ( including deep networks ) , and dominate existing methods in our experimental evaluation . Finally , we combine the two in an iterated Transform-Estimate-Discard ( TED ) n framework that significantly improves both MPE estimation error and classifier error . We build on label shift methods [ 29 , 3 , 2 , 34 , 17 ] , that leverage black-box classifiers to reduce dimensionality , estimating the target label distribution as a functional of source and target pushforward distributions . While label shift methods rely on classifiers trained to separate previously seen classes , BBE is able to exploit a Positive-versus-Unlabeled ( PvU ) target classifier , which gives each input a score indicating how likely it is to be a positive sample . In particular , BBE identifies a threshold such that by estimating the ratio between the fractions of positive and unlabeled points receiving scores above the threshold , we obtain the mixture proportion α. BBE works because in practice , for many datasets , PvU classifiers , even when uncalibrated , produce outputs with near monotonic calibration diagrams . Higher scores correspond to a higher proportion of positives , and when the positive data contains a separable sub-domain , i.e. , a region of the input space where only the positive distribution has support , classifiers often exhibit a threshold above which the top bin contains mostly positive examples . We show that the existence of a ( nearly ) pure top bin is sufficient for BBE to produce a ( nearly ) consistent estimate pα , whose finite sample convergence rates depend on the fraction of examples in the bin and whose bias depends on the purity of the bin . Crucially , we can estimate the optimal threshold from data . We conduct a battery of experiments both to empirically validate our claim that BBE ’ s assumptions are mild and frequently hold in practice , and to establish the outperformance of BBE , CVIR , and ( TED ) n over the previous state of the art . We first motivate BBE by demonstrating that in practice PvU classifiers tend to isolate a reasonably large , reasonably pure top bin . We then conduct extensive experiments on semi-synthetic data , adapting a variety of binary classification datasets to the PU learning setup and demonstrating the superior performance of BBE and PU-learning with the CVIR objective . Moreover , we show that ( TED ) n , which combines the two in an iterative fashion , improves significantly over previous methods across several architectures on a range of image and text datasets . 2 Related Work . Research on MPE and PU learning date to [ 9 , 8 , 27 ] ( see review by [ 5 ] ) . Elkan and Noto [ 16 ] first proposed to leverage a PvU classifier to estimate the mixture proportion . Du Plessis and Sugiyama [ 13 ] propose a different method for estimating the mixture coefficient based on Pearson divergence minimization . While they do not require a PvU classifier , they suffer the same shortcoming . Both methods require that the positive and negative examples have disjoint support . Our requirements are considerably milder . Blanchard et al . [ 6 ] observe that without assumptions on the underlying positive and negative distributions , the mixture proportion is not identifiable . Furthermore , [ 6 ] provide an irreducibility condition that identifies α and propose an estimator that converges to the true α . While their estimator can converge arbitrarily slowly , Scott [ 39 ] showed faster convergence ( Op1 { ? nq ) under stronger conditions . Unfortunately , despite its appealing theoretical properties Blanchard et al . [ 6 ] ’ s estimator is computationally infeasible . Building on Blanchard et al . [ 6 ] , Sanderson and Scott [ 38 ] and Scott [ 39 ] proposed estimating the mixture proportion from a ROC curve constructed for the PvU classifier . However , when the PvU classifier is not perfect , these methods are not clearly understood . Ramaswamy et al . [ 35 ] proposed the first computationally feasible algorithm for MPE with convergence guarantees to the true proportion . Their method KM , requires embedding distributions onto an RKHS . However , their estimator underperforms on high dimensional datasets and scales poorly with large datasets . Bekker and Davis [ 4 ] proposed TIcE , hoping to identify a positive subdomain in the input space using decision tree induction . This method also underperforms in high-dimensional settings . In the most similar works , Jain et al . [ 21 ] and Ivanov [ 20 ] explore dimensionality reduction using a PvU classifier . Both methods estimate α through a procedure operating on the PvU classifier ’ s output . However , neither methods has provided theoretical backing . [ 20 ] concede that their method often fails and returns a zero estimate , requiring that they fall back to a different estimator . Moreover while both papers state that their method require the Bayes-optimal PvU classifier to identify α in the transformed space , we prove that even when hypothesis class is well specified for PvN learning , PvU training can fail to recover the Bayes-optimal scoring function . Furthermore , we also show that the heuristic estimator in Scott [ 39 ] can be thought of as using PvU classifier for dimensionality reduction . While this heuristic is similar to our estimator in spirit , we show that the functional form of their estimator is different from ours and note that their heuristic enjoys no theoretical guarantee . By contrast , our estimator BBE is theoretically coherent under mild conditions and outperforms all of these methods empirically . Given α , Elkan and Noto [ 16 ] propose a transformation via Bayes rule to obtain the PvN classifier . They also propose a weighted objective , with weights given by the PvU classifier . Other propose unbiased risk estimators [ 14 , 11 ] which require the mixture proportion α . Du Plessis et al . [ 14 ] proposed an unbiased estimator with non-convex loss functions satisfying a specific symmetric condition , and subsequently Du Plessis et al . [ 11 ] generalized it to convex loss functions ( denoted uPU in our experiments ) . in our experiments . Noting the problem of overfitting in modern overparameterized models , Kiryo et al . [ 23 ] propose a regularized extension that clips the loss on unlabeled data to zero . This is considered the current state-of-the-art in PU literature ( denoted nnPU in our experiments ) . More recently , Ivanov [ 20 ] proposed DEDPUL , which finetunes the PvU classifiers using several heuristics , Bayes rule , and Expectation Maximization ( EM ) . Since their method only applies a post-processing procedure , they rely on a good domain discriminator classifier in the first place and several hyperparameters for their heuristics . Several classical methods attempt to learn weights that identify reliable negative examples [ 30 , 28 , 26 , 31 , 44 ] . However , these earlier methods have not been successful with modern deep learning models . Algorithm 1 Best Bin Estimation ( BBE ) input : Validation positive ( Xp ) and unlabeled ( Xu ) samples . Blackbox model classifier pf : X Ñ r0 , 1s . Hyperparameter 0 ă δ , γ ă 1 . 1 : Zp , Zu “ fpXpq , fpXuq . 2 : pqupzq , pqppzq “ ř ziPZp Irziězs np , ř ziPZu Irziězs nu for all z P r0 , 1s . 3 : Estimate pc : “ argmincPr0,1s ˆ pqupcq pqppcq ` 1 ` γ pqppcq ˆ b logp4 { δq 2nu ` b logp4 { δq 2np ˙˙ . output : pα : “ pquppcq pqpppcq

Summary: This paper tackles the well known problem of mixture proportion estimation and Positive-Unlabelled (PU) learning. The authors propose a a (new) estimator for MPE based on the assumption of a PU classifier having a "pure" top bin. The authors then develop a novel  iterative outlier-removal-like algorithm for learning a classifier separating the positives and negatives from PU data assuming access to the true MPE. Finally, they combine these two approaches to create a novel alternating algorithm that estimates MPE based on a current PvN classifier, and use the MPE estimate to learn a better PvN classifier from PU data.