Imbalanced Classes: Part 1. For a recent data science project, I... | by Becca R | Towards Data Science

For a recent data science project, I developed a supervised learning model to classify the booking location of a first-time user of the vacation home site Airbnb. This dataset is available on Kaggle as a part of a 2015 Kaggle competition.
For my project, I decided to group users into two groups: those who booked their first trip within the U.S.A. and Canada, and those who booked their first trip elsewhere internationally, essentially turning the problem into a binary classification problem. Sounds simple, right?
The problem was that the classification target (booking location) is highly imbalanced. Nearly 75 percent of first time users booked trips within the U.S.A. and Canada. After my initial model showed promising results, further inspection of the model’s performance metrics highlighted a crucial problem when trying to perform binary classification with imbalanced class sizes. This post aims to highlight some pitfalls to be aware of when building a classification model with imbalanced classes, and highlights some methods to deal with these issues.
The data
This plots shows the significant imbalance inherent in my target class:
import matplotlib.pyplot as pltimport pandas as pdimport numpy as npimport pickleimport seaborn as snsdf = pd.read_pickle('data_for_regression_20k.pkl')sns.set_style("white")dests2 = df.groupby('country_USA_World_bi').agg({'country_USA_World_bi':['count']}).reset_index()dests2.columns = ['dest', 'count']dests2['pct'] = dests2['count']*100/(sum(dests2['count']))x = dests2['dest']y = dests2['pct']palette = ['olive','mediumvioletred']fig, ax = plt.subplots(figsize = (8,4))fig = sns.barplot(y, x, estimator = sum, ci = None, orient='h', palette=palette)y_lab = ['USA/Canada', 'International']ax.set_yticklabels(labels=y_lab, ha='right')for i, v in enumerate(y):    ax.text(v - 15, i + .05, str(int(v)+.5)+'%', color='white', fontweight='bold')plt.title('Country Destinations as Percent of Total Bookings',size = 16, weight = 'bold')plt.ylabel('Country')plt.xlabel('Percent of total');
Before fitting a logistic regression classifier to my data, I split the data into a training set (80%) and hold-out set for testing (20%). Due to the underrepresentation of international travelers, I used the stratify parameter to ensure that both target classes were represented in the test set. Then, I standardized both the train and test feature set using a standard scalar.
from sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import StandardScalerfrom sklearn.metrics import precision_score, recall_score, precision_recall_curve,f1_score, fbeta_score, make_scorery = df['country_USA_World_bi']  X = df.drop(['country_dest_id','country_USA_World_bi','month_created', 'day_created', 'month_active', 'day_active'], axis = 1)Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.2, stratify=y,random_state = 88)std_scale = StandardScaler()X_train_scaled = std_scale.fit_transform(Xtrain)X_test_scaled = std_scale.transform(Xtest)
The first model I ran was logistic regression, because logistic regression includes feature coefficients (which helps with interpretability). I fit an initial model with default hypterparamaters, and was pleasantly surprised to see that the model was 75 percent accurate prior to any hypterparameter tuning. Naturally, I found myself wondering if this was too good to be true. Remembering how 75 percent of users traveled within the U.S.A./Canada, couldn’t I be 75 percent accurate by simply guessing this destination every time?
In fact, model precision (the proportion of predicted international bookings that are actually international), model recall (the proportion of international bookings correctly identified by the model), and the f1 score (a balance of the two) are pretty poor.
For the purpose of this project, I am most interested in the recall score, since I think it is most useful for the model to be able to accurately predict users who will travel internationally (since those who travel internationally are more likely to be avid travelers with larger travel budgets). However, given the initial recall score of just 0.01, I had a long way to go to improve this model!
from sklearn.linear_model import LogisticRegressiondef fit_logistic_regression_classifier(X_training_set, y_training_set):    logreg = LogisticRegression(random_state=88)    model = logreg.fit(X_training_set, y_training_set)    y_pred = model.predict(X_test_scaled)    print('accuracy = ',model.score(X_test_scaled, ytest).round(2),          'precision = ',precision_score(ytest, y_pred).round(2),           'recall = ',recall_score(ytest, y_pred).round(2),           'f1_score = ',f1_score(ytest, y_pred).round(2)         )    return(y_pred)y_pred = fit_logistic_regression_classifier(X_train_scaled, ytrain)
A confusion matrix is a great tool to visualize the extent to which the model got, well, confused. The confusion matrix in sklearn gives raw value counts for the number of observations predicted to be in each class, by their actual class. The plot_confusion_matrix() function gives a visual representation of the percent of values in each actual and predicted class.
import itertoolsfrom sklearn.metrics import confusion_matrixdef make_confusion_matrix(cm, classes,title='Confusion matrix',cmap=plt.cm.Blues):    print(cm)    # Normalize values    cm = cm.astype('float')*100 / cm.sum(axis=1)[:, np.newaxis]    plt.imshow(cm, interpolation='nearest', cmap=cmap)    plt.title(title)    plt.colorbar()    tick_marks = np.arange(len(classes))    plt.xticks(tick_marks, classes, rotation=45)    plt.yticks(tick_marks, classes)fmt = '.2f'    thresh = cm.max() / 2.    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):        plt.text(j, i, format(cm[i, j], fmt),                 horizontalalignment="center",                 color="white" if cm[i, j] > 50 else "black")    plt.ylabel('True label')    plt.xlabel('Predicted label')    plt.tight_layout()def plot_confusion_matrix(y_test_set, y_pred):    class_names = ['Domestic','International']    cnf_matrix = confusion_matrix(y_test_set, y_pred)    np.set_printoptions(precision=2)    plt.figure()    make_confusion_matrix(cnf_matrix, classes=class_names, title='Confusion matrix with normalization');
The figure shows that my intuition was correct — the model had classified almost 100 percent of observations as domestic travelers, therefore hitting the mark 75 percent of the time!
plot_confusion_matrix(ytest, y_pred)
1. Random Oversampling
The Imbalanced Learn library includes a variety of methods to rebalance classes for more accurate predictive capability. The method I tried is called Random Oversampling. According to the documentation, “random over-sampling can be used to repeat some samples and balance the number of samples between the dataset.” Basically, this rebalancing method uses random sampling of target classes with replacement to get a balanced representation of each class in the training set. Indeed, after fitting the Random Over Sampler to my training set, I had a sample of 12,743 observations in each target class, compared with my baseline scenario of 3,186 domestic bookings and 1,088 international bookings.
from imblearn.over_sampling import RandomOverSamplerros = RandomOverSampler(random_state=88)X_resampled, y_resampled = ros.fit_sample(X_train_scaled, ytrain)yvals, counts = np.unique(ytest, return_counts=True)yvals_ros, counts_ros = np.unique(y_resampled, return_counts=True)print('Classes in test set:',dict(zip(yvals, counts)),'\n',      'Classes in rebalanced test set:',dict(zip(yvals_ros, counts_ros)))
Just as before, I fit a logistic regression classifier with default parameters and observed the model’s performance metrics and confusion matrix. Since the model could no longer correctly guess the domestic location 75 percent of the time, its performance decreased significantly: accuracy dropped to 54 percent.
However, recall, my metric of interest, increased — from 0.01 to 0.58. Providing balanced class sizes in the test set significantly increased the model’s ability to predict the minority class (in my case, Airbnb bookings in international locations).
y_pred_ros = fit_logistic_regression_classifier(X_resampled, y_resampled)plot_confusion_matrix(ytest, y_pred_ros)
2. SMOTE and ADASYN
The Synthetic Minority Oversampling Technique (SMOTE) and the Adaptive Synthetic (ADASYN) are two additional methods for oversampling the minority class. Unlike Random Over Sampling, which over samples existing observations, SMOTE and ADASYN use interpolation to create new observations near existing observations of the minority class.
For the SMOTE rebalancing, I used the SMOTENC object, since the majority (all but six) of my features are non-continuous (i.e. categorical) features. Just as before, I ended up with a balanced training set.
ADASYN gave me a new training set that was around 49 percent travelers to U.S.A./Canada and 51 percent travelers abroad. This (trivial) imbalance is due to the way in which ADASYN creates new data points around difficult to classify points according to a weighted distribution of their difficulty (see He, et al., 2008).
Just like with Random Over Sampling, the model’s ability to classify all destinations (accuracy) decreased with oversampling. On the other hand, the model’s ability to classify the minority class (recall) improved with both SMOTE and ADASYN.
from imblearn.over_sampling import SMOTENC, ADASYNsmote_nc = SMOTENC(categorical_features=list(np.arange(7,80)), random_state=88)X_smoted, y_smoted = smote_nc.fit_resample(X_train_scaled, ytrain)adasyn = ADASYN(random_state=88)X_adasyn, y_adasyn = adasyn.fit_resample(X_train_scaled, ytrain)yvals, counts = np.unique(ytest, return_counts=True)yvals_smt, counts_smt = np.unique(y_smoted, return_counts=True)yvals_ads, counts_ads = np.unique(y_adasyn, return_counts=True)print('Classes in test set:',dict(zip(yvals, counts)),'\n',      'Classes in rebalanced test set with SMOTENC:',dict(zip(yvals_smt, counts_smt)),'\n',      'Classes in rebalanced test set with ADASYN:',dict(zip(yvals_ads, counts_ads)))
y_pred_smt = fit_logistic_regression_classifier(X_smoted, y_smoted)plot_confusion_matrix(ytest, y_pred_smt)
y_pred_ads = fit_logistic_regression_classifier(X_adasyn, y_adasyn)plot_confusion_matrix(ytest, y_pred_ads)
3. Gridsearch on balanced classes
Since the baseline model with ADASYN oversampling performed best in terms of recall, I performed a gridsearch on this test set to find the parameters that would furhter optimize model performance.
from sklearn.model_selection import GridSearchCVgrid = {"C":np.logspace(-3,3,7), "penalty":["l1","l2"]}# l1 lasso l2 ridgelogreg = LogisticRegression(random_state=88)logreg_cv = GridSearchCV(logreg,grid,cv=5,scoring='recall')logreg_cv.fit(X_adasyn, y_adasyn)print("tuned hpyerparameters :(best parameters) ", logreg_cv.best_params_)
The logistic regression model with a C parameter of 0.001 and a L2 regularization penalty had an improved recall score of 0.65. This means that the model was able to effectively catch 65 percent of new-users who would book Airbnbs internationally.
y_pred_cv = logreg_cv.predict(X_test_scaled)print('accuracy = ',logreg_cv.score(X_test_scaled, ytest).round(2),        'precision = ',precision_score(ytest, y_pred_cv).round(2),         'recall = ',recall_score(ytest, y_pred_cv).round(2),         'f1_score = ',f1_score(ytest, y_pred_cv).round(2)        )plot_confusion_matrix(ytest, y_pred_cv)
While balanced classes and hyperparameter tuning yielded significant improvements to the model’s recall score, model precision remained quite low, at 0.3. This means that only 30% of users classified as international travellers are actually booking Airbnbs internationally. In a business setting, a model like this might be used to inform targeted ads for vacation homes based on predicted booking destination. This means that 70 percent of users receiving suggestions for, say, homes overlooking the Eiffel Tower will in fact be looking to travel domestically. Such mis-targeting would not only prove irrelevant to this group, but failure to disseminate relevant ads to the U.S.A./Canada group could mean missed revenue over time.
Now that I’ve accounted for overestimation of model performance by oversampling the minority class, next steps might include additional feature engineering to tease out more signal and fitting alternative classification algorithms (such as K-Nearest Neighbors or Random Forest Classifier).
Conclusion
In this example, model accuracy declined significantly once I rebalanced the target class sizes. Even after hypterparameter tuning using gridsearch cross-validation, the logistic regression model was 10 percentage points less accurate than the baseline model with imbalanced classes.
This example demonstrates the importance of taking class imbalance into account so as to avoid over-estimating the accuracy of classification models. I have also outlined, with working code, three techniques for re-balancing classes through over-sampling (Random Over Sampling, SMOTE, and ADASYN). Further information on each technique can be found on the Imbalanced-Learn documentation.