Semi-Supervised Classification of Unlabeled Data (PU Learning) | by Alon Agmon | Towards Data Science

Suppose you have a dataset of payment transactions. Some of the transactions are labeled as fraud and the rest are labeled as authentic, and you are required to design a model that will distinguish between fraudulent and authentic transactions. Assuming you have enough data and good features, this seems like a straightforward classification task. However, suppose that only 15% of your data is labeled, and that the labeled samples belong to just one class so that your training set consists of 15% samples labeled as authentic while the rest are unlabeled and could be either authentic or fraudulent. How would you go about classifying them? Has this twist in the requirements just turned this task into an unsupervised learning problem? Well, not necessarily.
This problem — which is often referred to as a PU (positive and unlabeled) classification problem — should be first distinguished from two similar and common “labeling issues” that complicate many classification tasks. The first and most common type of labeling issue is the problem of a small training set. It arises when although you have a decent amount of data, just a small part of it is actually labeled. This problem has many varieties and quite a few specific training methodologies. Another common labeling issue (that is often conflated with PU problems) involves cases in which our training data set is fully labeled, but it consists of just one class. Suppose, for example, that all we have is a data set of non-fraudulent transactions and that we need to use this data set to train a model to distinguish between (similar) non-fraudulent transactions and fraudulent ones. This is also a common problem that is usually treated as an unsupervised outlier detection problem, though there are also quite a few tools widely available in the ML landscape that are specifically designed to handle these scenarios (OneClassSVM might be the most famous).
A PU classification problem, in contrast, is a case involving a training set in which just part of the data is labeled as positive while the rest is unlabeled and could be either positive or negative. For instance, suppose that your employer is a bank that can provide you with a lot of transactional data, but can only confirm that part of it is 100% authentic. The example that I will use here involves a similar scenario with respect to fraudulent banknotes. It includes a data set of 1200 banknotes, most of which are unlabeled while just part of them is confirmed as authentic. Although PU problems are also quite common, they are often much less discussed than the two classification problems mentioned earlier, and very few hands-on examples or libraries are widely available.
The purpose of this post is to present one possible approach to PU problems which I have recently used in a classification project. It is based on the paper “Learning classifiers from only positive and unlabeled data” (2008) written by Charles Elkan and Keith Noto, and on some code written by Alexandre Drouin. Although there are more approaches to PU learning in scientific publications (I intend to discuss another rather popular approach in a future post), Elkan and Noto’s (E&N) approach is quite simple and can be easily implemented in Python.
E&N essentially claim that given a data set in which we have positive and unlabeled data, the probability that a certain sample is positive [ P(y=1|x)] equals the probability that the sample is labeled [P(s=1|x)] divided by the probability that a positive sample is labeled in our data set [P(s=1|y=1)].
If this claim is true (and I’m not going to prove or defend it — you can read the proof in the paper itself and experiment with the code), then it seems relatively easy to implement. This is so because although we don't have enough labeled data to train a classifier to tell us whether a sample is positive or negative, in a PU scenario we do have enough labeled data to tell us whether a positive sample is likely to be labeled or not and, according to E&N, this is enough to estimate how likely it is to be positive.
Putting things more formally, given an unlabeled data set with just a group of samples labeled as positive, we can estimate the probability that unlabeled sample x is positive if we estimate P(s=1|x) / P(s=1|y=1). Luckily, we can use almost any sklearn-based classifier to estimate this according to the following steps:
(1) Fit a classifier on a data set containing labeled and unlabeled data while using an isLabeled indicator as a target y. Fitting a classifier in this way will train it to predict the probability that a given sample x is labeled — P(s=1| x).
(2) Use the classifier to predict the probability that the known positive samples in our data set are labeled so that the predicted results will represent the probability that a positive sample is labeled — P(s=1|y=1|x)
Calculate the mean of these predicted probabilities and that will be our P(s=1|y=1).
Having estimated P(s=1|y=1), all we need to do in order to predict the probability that data point k is positive according to E&N is to estimate P(s=1|k) or the probability that it is labeled which is exactly what the classifier we trained on (1) knows how to do.
(3) Use the classifier we trained on (1) to estimate the probability that k is labeled or P(s=1|k).
(4) Once we have estimated P(s=1|k), we can actually classify k by dividing it by P(s=1|y=1), which has been estimated on step (2), and get the actual probabilities that it belongs to either class.
Steps 1–4 above can be implemented as follows:
# prepare datax_data = the training sety_data = target var (1 for the positives and not-1 for the rest)# fit the classifier and estimate P(s=1|y=1)classifier, ps1y1 =        fit_PU_estimator(x_data, y_data, 0.2, Estimator())# estimate the prob that x_data is labeled P(s=1|X)predicted_s = classifier.predict_proba(x_data)# estimate the actual probabilities that X is positive# by calculating P(s=1|X) / P(s=1|y=1)predicted_y = estimated_s / ps1y1
Let’s start with the main move here: fit_PU_estimator() method.
The fit_PU_estimator() method completes 2 main tasks: it fits a classifier you choose on a sample of the positive and unlabeled training set and then estimates the probability that a positive sample is labeled. Correspondingly, it returns the fitted classifier (that learned to estimate the proba that a given sample is labeled) and the estimated probability P(s=1|y=1). After that, all we need to do is find P(s=1|x) or the probability that x is labeled. Because that's what our classifier is trained to do, we just need to call its predict_proba() method. Finally, in order to actually classify sample x we just need to divide the result by P(s=1|y=1) that we have already found. This can be represented in code as:
The implementation of the fit_PU_estimator() method itself is quite self-explanatory :
In order to test this, I used the Bank Note Authentication data set, which is based on 4 data points that were extracted from images of genuine and forged banknotes. I first used the classifier on the labeled data set in order to set a baseline, and then removed the labels of 75% of the samples in order to test how it performs on a P&U data set. As the output shows, it is true this data set is not one of the hardest to classify, but you can see that although the PU classifier only “knew” about 153 positive samples while all the rest 1219 were unlabeled, it performed quite well compared to a classifier that had all labels available. However, it did lose about 17% of the recall and therefore lost quite a few true positives. Yet, whenever this is all we have, then I believe that these results are quite satisfying comparing to the alternatives.
===>> load data set <<===data size: (1372, 5)Target variable (fraud or not):0    7621    610===>> create baseline classification results <<===Classification results:f1: 99.57%roc: 99.57%recall: 99.15%precision: 100.00%===>> classify on all the data set <<===Target variable (labeled or not):-1    12191     153Classification results:f1: 90.24%roc: 91.11%recall: 82.62%precision: 99.41%
Few important notes. First, the performance of this approach greatly depends on the size of the dataset. In this example, I have used about 150 positive samples and about 1200 unlabeled. This is far from being the ideal data set for this approach. If we only had 100 samples, for example, our classifier would have performed very poorly. Second, as the attached notebook shows, there are a few variables to tune (such as the size of the sample to be set aside, the probability threshold to use for classification, etc), but the most important one is probably the chosen classifier and its parameters. I have chosen to use XGBoost because it performs relatively well on small data sets with few features, but it is important to note that it will not perform best in every scenario and it is important to test for the right classifier.
The notebook is available here.