Probability Calibration for Imbalanced Dataset | by Kyosuke Morita | Towards Data Science

When we are trying to build a machine learning model for real-world problems, we are often faced with imbalanced datasets. Resampling method, especially undersampling is one of the most widely used methods to overcome the class imbalance (I also showed how those resampling methods worked on my Master dissertation in the other medium post). However, implementing this method tends to increase false positive due to the different class distribution in train and test set. This biases the classifier and increases false positive. [1] Pozzolo, et al., (2015) argues that we can correct the bias due to the undersampling by using Bayes Minimum Risk theory. This helps us to find the correct classification threshold.
How does Probability Calibration work?ExperimentConclusion
How does Probability Calibration work?
Experiment
Conclusion
As briefly mentioned above, undersampling causes a bias in the posterior probabilities. This is due to the characteristic of random undersampling, which downsizes the majority class by removing them randomly until both classes have the same number of observations. This makes the class distribution of the training set different from the one in the test set. So how exactly probability calibration using Bayes Minimum Risk theory works on this problem? — the basic idea of this method is trying to reduce/remove the bias caused by random undersampling by taking into the undersampling ratio β account. Let’s have a look into some definitions:
Let ps be the probability of the prediction being a positive class after random undersampling;
, and p be the probability of the prediction given features (unbalanced). We can write ps as a function of p;
, where β is a probability of selecting negative class with undersampling, which can be expressed below.
, which can be written
The equation above can be solved for p and expressed as below.
So after applying undersampling ratio β, we can calculate for p, which is the unbiased probability.
The threshold for this is can be;
, which is a probability of a positive class in a dataset.
This was a brief introduction of probability calibration by using Bayes Minimum Risk theory. Now we will go on to see how this works in an example with codes.
In this section, we will see how the probability calibration technique the model performance on a binary classification problem on the famous credit card fraud dataset on Kaggle. This dataset consists of 28 PCA features (all of them are anonymous) plus the Amount feature. The target feature is binary, either fraud or not. The positive class is 0.17% of a whole dataset, which is severely imbalanced. Let’s go through the example with codes.
First, import the packages.
## configimport numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snsimport io, os, sys, types, gc, refrom sklearn import preprocessingfrom sklearn.linear_model import LogisticRegressionfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import average_precision_score,confusion_matrix, precision_score, recall_score, precision_recall_curve, f1_score, log_lossfrom sklearn.decomposition import PCApca = PCA(n_components=1,random_state=42)from imblearn.under_sampling import RandomUnderSamplerrus = RandomUnderSampler(random_state=42)from imblearn.ensemble import BalancedBaggingClassifierfrom src.functionalscripts.BMR import *def make_prediction(model,X,threshold):    y_pred = model.predict_proba(X)    y_predicted = np.where(y_pred[:,1]>=threshold,1,0)    return y_pred, y_predicteddef evaluation(true, pred):        print('F1-score: ' + str(round(f1_score(true,pred),4)), '\n''Precision: ' + str(round(precision_score(true,pred),4)), '\n''Recall: ' + str(round(recall_score(true,pred),4)), '\n''Log loss: ' + str(round(log_loss(true,pred),4)), '\n''Cohen-Kappa: ' + str(round(cohen_kappa_score(true,pred),4)), '\n''Confusion matrix:' + '\n' + str(confusion_matrix(true,pred)))
Now read the dataset.
# read the datadf = pd.read_csv('src/resources/data/creditcard.csv')
Here are the first few rows of the dataset.
Class column is our target variable and Amount column is the transaction amount. Now see the ratio of the positive class.
# The percentage of positive class in this datasetlen(df[df['Class']==1])/len(df)
The positive class is as I mentioned above, 0.17%. Now we will move on to the model building. We will use logistic regression just to see. Let’s normalise the Amount column and split into train and test dataset.
# Normalise the Amount featuredf['amt'] = preprocessing.normalize(np.array(df['Amount']).reshape(-1,1),norm='l2')
Now, the data are ready, let’s split into train (80% of whole data) and test (20% of whole data) dataset and implement random undersampling.
# Split the dataset into train and test, and drop unnecessary featurestr, te = train_test_split(df.drop(['Amount','Time'],1), test_size=.2,random_state=42)# Rundom Under Sampling (RUS)tr_x_rus, tr_y_rus = rus.fit_resample(tr.drop(['Class'],1),tr.Class)
We can check the distribution of classes in train, test and train set after random undersampling.
# See the class distribution with featuresfeats_distribution_rus = pd.DataFrame(pca.fit_transform(tr_x_rus),columns=['after']).reset_index(drop=True)feats_distribution_rus['Class'] = tr_y_russns.regplot(x='after',y='Class',data=feats_distribution_rus,logistic=True, n_boot=500, y_jitter=.03)plt.title('Class distribution of training set after undersampling')plt.show()
We have the distribution of train set in the left upper corner, distribution after random undersampling in the left down corner and right-hand side are both test set (for comparison). y-axis shows the class and 1 means the transaction was a fraud and otherwise not a fraud. We can see similar distributions in train and test set as they were created by random sampling. On the other hand, in the train set after random undersampling demonstrates quite different distribution from others. Let’s see how this difference impact on the classifiers.
Now build models. For the comparison, let’s see the performance of RUS bagging (Random Undersampling + bagging) as well.
# Logistic regressionlogit = LogisticRegression(random_state=42,solver='lbfgs',max_iter=1000)# Rundom Under Sampling (RUS)tr_x_rus, tr_y_rus = rus.fit_resample(tr.drop(['Class'],1),tr.Class)logit_rus = logit.fit(tr_x_rus,tr_y_rus)# RUS baggingbc = BalancedBaggingClassifier(base_estimator=logit,random_state=42)logit_bc = bc.fit(tr.drop(['Class'],1),tr.Class)
Now we implement the probability calibration method using Bayes Minimum Risk. Here we create beta (minority selection ratio), tau (threshold) and calibration functions.
# BMR (Bayes Minimum Risk) implementation# Pozzolo et al., 2015, Calibrating Probability with Undersamplingclass BMR:    def beta(binary_target):        return binary_target.sum()/len(binary_target)    def tau(binary_target, beta):        return binary_target.sum()/len(binary_target)    def calibration(prob, beta):        return prob/(prob+(1-prob)/beta)
Apply those calibration techniques to both predicted probabilities by RUS and RUS Bagging.
# Calibrationbeta = BMR.beta(tr.Class)tau = BMR.tau(tr.Class,beta)# with RUSy_pred_calib_rus = BMR.calibration(prob=logit_rus.predict_proba(te.drop(['Class'],1))[:,1],beta=beta)y_predicted_calib_rus = np.where(y_pred_calib_rus>=tau,1,0)# wtih RUS baggingy_pred_calib_bc = BMR.calibration(prob=logit_bc.predict_proba(te.drop(['Class'],1))[:,1],beta=beta)y_predicted_calib_bc = np.where(y_pred_calib_bc>=tau,1,0)
Now we have all the predictions, let’s evaluate them and see how are their performance. I set the thresholds for RUS and RUS bagging model as 0.5.
# Evaluation## Random Under Sampling (RUS)y_pred_rus, y_predicted_rus = make_prediction(model=logit_rus, X=te.drop(['Class'],1), threshold=.5)evaluation(te.Class, y_predicted_rus)## RUS + Baggingy_pred_bc, y_predicted_bc = make_prediction(model=logit_bc, X=te.drop(['Class'],1), threshold=.5)evaluation(te.Class, y_predicted_bc)## Calibration with Rundom undersamplingevaluation(te.Class, y_predicted_calib_rus)## Calibration with RUS baggingevaluation(te.Class, y_predicted_calib_bc)
Here are the results. As we expected, there are so many false positives, which we can see from the precision scores.
Oh wait, probability calibration didn’t change anything at all? There must be something wrong. Let’s see the prediction distributions of predicted probability of RUS Bagging with calibration. The blue verticle line is the mean of predicted probability by RUS Bagging with calibration and red verticle line is the mean of predicted probability by RUS Bagging model. Obviously their means are quite far away, for calibrated probability mean is 0.0021 and before calibration is 0.5. Considering the positive class exists 0.17% in a whole dataset, the calibrated probability seems quite close to the actual distribution.
So if we modify the threshold, it should work better and here we can see the results with modified thresholds. Thresholds before calibration and after calibration on RUS model are set at 0.99 and calibration with RUS Bagging is set at 0.8.
As we can see, after calibration those scores improved, especially the difference between before and after calibration on the random undersampling model are significant.
So by correcting the biased probability by using probability calibration, we could see the performance improvement.
3. Conclusion and thoughts
In this blog post, we went through Pozzolo, et al. (2015). This answers our experience that having more false positive after applying random undersampling. It was quite interesting to see how the resampling method biases the distribution. This would be causing a similar problem in time-series problems as the target variable we are predicting can be quite different from when we trained the model.
Would be interesting to see if this sort of method works for different types of classifiers like tree-based or neural network and also different types of resampling methods.
The codes for this post is available on my GitHub page.
Introduced the resampling causes a bias in a posterior distribution
Introduced probability calibration method by using Bayes Minimum Risk theory (Pozzoli, et al. 2015)
Showed the example of this method
Confirmed that this method corrects the bias and improve the model results
[1] Pozzolo, et al., Calibrating Probability with Undersampling for Unbalanced Classification (2015), 2015 IEEE Symposium Series on Computational Intelligence