Machine Learning Resampling Techniques for Class Imbalances | by Allison Kelly | Towards Data Science

Let’s face it. Inequality sucks. And I’m not even talking about the fact that African Americans and Hispanics make up 56% of the American prison population despite being 32% of the total population or that the combined wealth of Bill Gates, Jeff Bezos, and Warren Buffet is more than the combined wealth of the bottom 50% of Americans.
As much as I’d like to rage about those facts, for now I’m talking about class imbalance in the context of machine learning classification models. According to Wikipedia, algorithmic bias “can emerge due to many factors, including but not limited to the design of the algorithm or the unintended or unanticipated use or decisions relating to the way data is coded, collected, selected or used to train the algorithm.” Training your model with highly imbalanced data can favor the majority class which can have serious implications.
Take Kaggle’s Cervical Cancer Risk Classification dataset.
Only 2% of all diagnoses included in the dataset were found to be cancer. After a simple Random Forest Classification model was performed, the accuracy of the model was 99%!
from sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import accuracy_score, recall_score, classification_reportfrom sklearn.model_selection import train_test_splitX = df.drop('Cancer', axis=1).dropna()y = df['Cancer'].dropna()X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25)random_forest = RandomForestClassifier(n_estimators=500).fit(X_train, y_train)y_pred = random_forest.predict(X_test)print('Accuracy score: ' + str(accuracy_score(y_test, y_pred)))print('Recall score: ' + str(recall_score(y_test, y_pred)))print(classification_report(y_test, y_pred))
That can’t quite be possible. Let’s dig a little deeper. Let’s only include two of the 36 columns in the model.
X = df.loc[:,['Smokes', 'Hormonal Contraceptives']]y = df['Cancer'].dropna()X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25)random_forest = RandomForestClassifier(n_estimators=500).fit(X_train, y_train)y_pred = random_forest.predict(X_test)print('Accuracy score: ' + str(accuracy_score(y_test, y_pred)))print('Recall score: ' + str(recall_score(y_test, y_pred)))print(classification_report(y_test, y_pred))
But the accuracy only dropped to 97%! There’s very little chance that the two features we chose randomly can predict cancer with 97% accuracy.
We already know that 2% of patients were diagnosed with cancer, but our second model predicts none of the patients would have cancer. See how this can become a problem?
In serious medical diagnosis models, punitive models such as COMPAS, the recidivism risk model, and fraud detection, there are human lives that hang in the balance. In these cases, it’s best to err on the side of caution and protect as many people as possible — whether that means lowering the number of false positives or false negatives.
When thinking about how to optimize these kinds of models, there are a few metrics we can interpret.
Recall: Taking the cervical cancer risk dataset as an example, you can ask yourself of the model — out of all the patients that were actually diagnosed with cancer, what percentage did our model predict as having cancer?
However, recall doesn’t give you the entire picture. Say your model categorized everyone as having cancer, including those who don’t, your score would be 100%. A low recall score is indicative of a high number of false negatives.
Precision: Precision asks the opposite question — out of all the patients the model predicted as having cancer, how many actually did have cancer?
If your model predicted 10 patients had cancer our score would be 100% if all of the predictions were correct, even if a thousand more patients went undiagnosed. A low precision score is indicative of a high number of false positives.
Accuracy: One of the more robust evaluation metrics is Accuracy, as it measures the total number of true predictions, both positive and negative. It’s the most common metric for classification tasks.
F1 Score: The F1 Score is another extremely informative metric. Because it measures the “harmonic mean of precision and recall,” it cannot be high without both precision and recall being high, indicative of an overall well-performing model. However, the F1 score can be formatted to account for binary, multiclass, and imbalanced classification problems with the following parameters in the sklearn.metrics.f1_score method:
Binary — To be used with binary classification problems.
Micro — Counts the total true positives, false negatives and false positives.
Macro — Calculates the unweighted mean of all classes (for multiclass problems.)
Weighted — Accounts for class imbalances by weighing the true positives for each class and taking the average score.
Samples — Finds the average score of each metric for each class.
Equipped with this information, we’ll optimize for the binary F1 score, since we only have two classes. You may be thinking “why not use the weighted F1 score if our data is so severely imbalanced?” That’s where resampling methods come in!
We’ll explore three methods (though there are many more out there) that are simple and useful — undersampling the majority, oversampling the minority, and SMOTE (synthetic minority oversamplting technique). Each method we’ll be using aims to create a training set with a 50–50 distribution since we’re working with a binary classification problem.These methods can be used to create a 25–25–25–25 distribution for a four class multi-class problem, regardless of the initial distribution of classes, or another ratio that may train your model with better results.
Make sure to split your data into training and testing sets BEFORE resampling! If you don’t, you’ll be compromising the quality of your model from data leakage, causing overfitting and poor generalization.
# Import the resampling packagefrom sklearn.utils import resample# Split into training and test setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25)# Returning to one dataframetraining_set = pd.concat([X_train, y_train], axis=1)# Separating classescancer = training_set[training_set.Cancer == 1]not_cancer = training_set[training_set.Cancer == 0]
Undersampling the Majority
Undersampling can be defined as reducing the number of the majority class. This technique is best used on data where you have thousands if not millions of datapoints. Typically, you wouldn’t want to reduce the amount of data you’re working with, but if you can sacrifice some training data, this technique will be useful. Here’s how it works on the cervical cancer dataset:
# Undersampling the majorityundersample = resample(not_cancer,                        replace=True,                        n_samples=len(cancer), #set the number of samples to equal the number of the minority class                       random_state=42)# Returning to new training setundersample_train = pd.concat([cancer, undersample])undersample_train.Cancer.value_counts(normalize=True)
We’ve got our evenly distributed classes! Now we can test it on the Random Forest classifier.
# Separate undersampled data into X and y setsundersample_x_train = undersample_train.drop('Cancer', axis=1)undersample_y_train = undersample_train.Cancer# Fit model on undersampled dataundersample_rf = RandomForestClassifier(n_estimators=500).fit(undersample_x_train, undersample_y_train)# Make predictions on test setsy_pred = random_forest.predict(X_test)print('Accuracy score: ' + str(accuracy_score(y_test, y_pred)))print('Average Recall score: ' + str(recall_score(y_test, y_pred, average='macro')))print(classification_report(y_test, y_pred))
Not exactly the best results, but because we reduced the number of the majority class, we trained our model on only 28 instances which is far too small of a sample size. Next, we’ll try oversampling the minority.
Oversampling the Minority
Oversampling the minority will increase the number of datapoints in the minority class, again aiming to evenly distribute the classes in the training set. We’ll repeat the same process as before.
# Oversampling the minorityoversample = resample(cancer,                        replace=True,                        n_samples=len(not_cancer), #set the number of samples to equal the number of the majority class                       random_state=42)# Returning to new training setoversample_train = pd.concat([not_cancer, oversample])oversample_train.Cancer.value_counts(normalize=True)
# Separate oversampled data into X and y setsoversample_x_train = oversample_train.drop('Cancer', axis=1)oversample_y_train = oversample_train.Cancer# Fit model on oversampled dataoversample_rf = RandomForestClassifier(n_estimators=500).fit(oversample_x_train, oversample_y_train)# Make predictions on test setsy_pred = oversample_rf.predict(X_test)print('Accuracy score: ' + str(accuracy_score(y_test, y_pred)))print('Average Recall score: ' + str(recall_score(y_test, y_pred, average='macro')))print(classification_report(y_test, y_pred))
Unfortunately, our results are only marginally better. We have one more technique to try.
SMOTE (synthetic minority oversampling technique)
SMOTE synthesizes datapoints from the existing pool of the minority class and adds them to the dataset. This technique ensures there’s very little data leakage by creating new, unseen datapoints for the model to train on.
# Import the SMOTE packagefrom imblearn.over_sampling import SMOTE# Synthesize minority class datapoints using SMOTEsm = SMOTE(random_state=42, sampling_strategy=’minority’)smote_x_train, smote_y_train = sm.fit_resample(X_train, y_train)# Separate into training and test setssmote_x_train = pd.DataFrame(smote_x_train, columns = X_train.columns)smote_y_train = pd.DataFrame(smote_y_train, columns = ['Cancer'])smote = RandomForestClassifier(n_estimators=1000).fit(smote_x_train, smote_y_train) # Predict on training setsmote_preds = smote.predict(X_test)# Checking accuracy and recallprint('Accuracy Score: ', accuracy_score(y_test, smote_preds),'\n\n')print('Averaged Recall Score: ', recall_score(y_test, smote_preds, average='macro'), '\n\n')    print(classification_report(y_test, smote_preds))
Accuracy and f1 score increased, however the recall score dropped slightly. Depending on your use case for the model, at this point you’ll have to decide which model protects the most people.
One of the best ways to visualize the quality of your model is by examining the ROC curve. ROC (Receiver Operating Characteristic) curves plot the true positive rate, TPR, against the false positive rate, FPR. The best prediction would be found at the point (0,1) where there are no false positives (costs) and 100% true positives (benefits).
Included in the plot is the line of no-descrimination which demonstrates random guessing, similar to the probability of a coin-flip. Points above the line are “good” guesses, as they are closer to a perfect outcome at the point (0,1). The opposite is also true, where points below the line indicate poor predictions.
AUC, or area under the curve, is a quantitative measure of the degree of separability. The closer the predictions are to the ideal outcome, the larger the AUC would be. When AOC tells us not to settle for less, she’s telling us to aim for an AUC close to 1 or 100%.
Here is an example with the Random Forest Classifier trained with the SMOTE resampled training set (read the documentation here):
y_score = smote.fit(smote_x_train, smote_y_train).predict_proba(X_test)[:,1]fpr, tpr, thresholds = roc_curve(y_test, y_score)print('AUC: {}'.format(auc(fpr, tpr)))
Not too bad! There’s still plenty of room for improvement, but we can see what this looks like in a plot:
plt.figure(figsize=(10, 8))lw = 2plt.plot(fpr, tpr, color=’darkorange’, lw=lw, label=’ROC curve’)plt.plot([0, 1], [0, 1], color=’navy’, lw=lw, linestyle=’ — ‘)plt.xlim([0.0, 1.0])plt.ylim([0.0, 1.05])plt.yticks([i/20.0 for i in range(21)])plt.xticks([i/20.0 for i in range(21)])plt.xlabel(‘False Positive Rate’)plt.ylabel(‘True Positive Rate’)plt.title(‘Receiver operating characteristic (ROC) Curve’)plt.legend(loc=’lower right’)plt.show()
Ideally, we’d like to see the orange line far steeper and not level out until it’s much closer to (0,1) but this is a great starting point.
As with all machine learning projects, the process is iterative. There are still other ways to resample and validate your model that should be explored before you decide which to move forward with. Think thoroughly about which performance metric fits your purpose, choose an algorithm that excels when it comes to your type of data, and evaluate your model based on the one that can do the least harm once it’s deployed.
Comment below and tell me other ways you’ve resample your data!