A common approach in positive-unlabeled learning is to train a classification model between labeled and unlabeled data. This strategy is in fact known to give an optimal classifier under mild conditions; however, it results in biased empirical estimates of the classifier performance. In this work, we show that the typically used performance measures such as the receiver operating characteristic curve, or the precisionrecall curve obtained on such data can be corrected with the knowledge of class priors; i.e., the proportions of the positive and negative examples in the unlabeled data. We extend the results to a noisy setting where some of the examples labeled positive are in fact negative and show that the correction also requires the knowledge of the proportion of noisy examples in the labeled positives. Using state-of-the-art algorithms to estimate the positive class prior and the proportion of noise, we experimentally evaluate two correction approaches and demonstrate their efficacy on real-life data. Introduction Performance estimation in binary classification is tightly related to the nature of the classification task. As a result, different performance measures may be directly optimized during training. When (mis)classification costs are available, the classifier is ideally trained and evaluated in a costsensitive mode to minimize the expected cost (Whalen 1971; Elkan 2001). More often, however, classification costs are unknown and the overall performance is assessed by averaging the performance over a range of classification modes. The most extensively studied and widely used performance evaluation in binary classification involves estimating the Receiver Operating Characteristic (ROC) curve that plots the true positive rate of a classifier as a function of its false positive rate (Fawcett 2006). The ROC curve provides insight into trade-offs between the classifier’s accuracies on positive versus negative examples over a range of decision thresholds. Furthermore, the area under the ROC curve (AUC) has a meaningful probabilistic interpretation that correlates with the ability of the classifier to separate classes and is often used to rank classifiers (Hanley and McNeil 1982). Another important performance criterion generally used in information retrieval relies on the precision-recall Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. (pr-rc) curve, a plot of precision as a function of recall. The precision-recall evaluation, including summary statistics derived from the pr-rc curve, may be preferred to ROC curves when classes are heavily skewed (Davis and Goadrich 2006). Although model learning and performance evaluation in a supervised setting are well understood (Hastie et al. 2001), the availability of unlabeled data gives additional options and also presents new challenges. A typical semi-supervised scenario involves the availability of positive, negative and (large quantities of) unlabeled data. Here, the unlabeled data can be used to improve training (Blum and Mitchell 1998) or unbias the labeled data (Cortes et al. 2008); e.g., to estimate class proportions that are necessary to calibrate the model and accurately estimate precision when class balances (but not class-conditional distributions) in labeled data are not representative (Saerens et al. 2002). This is often the case when it is more expensive or difficult to label examples of one class than the examples of the other. A special case of the semi-supervised setting arises when the examples of only one class are labeled. It includes open-world domains such as molecular biology where, for example, wet lab experiments determining a protein’s activity are generally conclusive; however, the absence of evidence about a protein’s function cannot be interpreted as the evidence of absence. This is because, even when the labeling is attempted, a functional assay may not lead to the desired activity for a number of experimental reasons. In other domains, such as social networks, only positive examples can be collected (such as ‘liking’ a particular product) because, by design, the negative labeling is not allowed. The development of classification models in this setting is often referred to as positiveunlabeled learning (Denis et al. 2005). State-of-the-art techniques in positive-unlabeled learning tackle this problem by treating the unlabeled sample as negatives and training a classifier to distinguish between labeled (positive) and unlabeled examples. Following Elkan and Noto (2008), we refer to the classifiers trained on a labeled sample from the true distribution of inputs, containing both positive and negative examples, as traditional classifiers. Similarly, we refer to the classifiers trained on the labeled versus unlabeled data as non-traditional classifiers. In theory, the true performance of both traditional and non-traditional classifiers can be evaluated on a labeled samar X iv :1 70 2. 00 51 8v 1 [ st at .M L ] 2 F eb 2 01 7 ple from the true distribution (traditional evaluation). However, this is infeasible for non-traditional learners because such a sample is not available in positive-unlabeled learning. As a result, the non-traditional classifiers are evaluated by using the unlabeled sample as substitute for labeled negatives (non-traditional evaluation). Surprisingly, for a variety of performance criteria, non-traditional classifiers achieve similar performance under traditional evaluation as optimal traditional classifiers (Blanchard et al. 2010; Menon et al. 2015). The intuition for these results comes from the fact that in many practical situations, the posterior distributions in traditional and non-traditional setting provide the same optimal ranking of data points on a given test sample (Jain et al. 2016; Jain, White, and Radivojac 2016). Furthermore, the widely-accepted evaluation approaches using ROC or pr-rc curves are insensitive to the variation of raw prediction scores unless they affect the ranking. Though the efficacy of non-traditional classifiers has been thoroughly studied (Peng et al. 2003; Elkan and Noto 2008; Ward et al. 2009; Menon et al. 2015), estimating their true performance has been much less explored. Such performance estimation often involves computing the fraction(s) of correctly and incorrectly classified examples from both classes; however, in absence of labeled negatives, the fractions computed under the non-traditional evaluation are incorrect, resulting in biased estimates. Figure 1 illustrates the effect of this bias by showing the traditional and nontraditional ROC curves on a handmade data set. Because some of the unlabeled examples in the training set are in fact positive, the area under the ROC curve estimated when the unlabeled examples were considered negative (nontraditional setting) underestimates the true performance for positive versus negative classification (traditional setting). This paper formalizes and evaluates performance estimation of a non-traditional classifier in the traditional setting when the only available training data are (possibly noisy) positive examples and unlabeled data. We show that the true (traditional) performance of such a classifier can be recovered with the knowledge of class priors and the fraction of mislabeled examples in the positive set. We derive formulas for converting the ROC and pr-rc curves from the nontraditional to the traditional setting. Using these recovery formulas, we present methods to estimate true classification performance. Our experiments provide evidence that the methods for the recovery of a classifier’s performance are sound and effective. Problem formulation Consider a binary classification problem from input x ∈ X to output y ∈ Y = {0, 1} in a positive-unlabeled setting. Let f be the true distribution over the input space X from which the unlabeled sample is drawn and let f1 and f0 be the distributions of the positive and negative examples, respectively. It follows that f can be expressed as a two-component mixture containing f1 and f0 as f(x) = αf1(x) + (1− α)f0(x), for all x ∈ X where α ∈ [0, 1) is the mixing proportion (positive class prior) giving the proportion of positives in f . ́ ́ ° ° AUC = 0.8000 AUC = 0.9375 B. Positive vs. unlabeled A. Data set: prediction scores and class labels C. Positive vs. negative 0.986 yes, as 1 0.943 no 0.863 yes, as 1 0.789 no 0.009 no 0.699 yes, as 1 0.473 no 0.211 no 1 1 1 0 1 0 0 0 Prediction Labeled True class label
Output: Recovering True Classifier Performance in Positive-Unlabeled Learning