Classification of unbalanced datasets | by Michael Kareev | Towards Data Science

Let’s imagine you have a dataset with a dozen features and need to classify each observation. It can be either a two-class problem (your output is either 1 or 0; true or false) or a multi-class problem (more than two alternatives are possible). In this case, however, there is a twist. The data is unbalanced. Think about patients who may or may not have cancer (the majority probably won’t) or a decision to extend a credit line (the majority of bank clients get an extension). Your machine learning algorithm will be “overexposed” to one class, and “underexposed” to another. There are numerous articles online about this problem with different approaches. Here I will combine a few of them to get a robust solution.
We’ll start with more straight-forward linear models, then add a couple of tweaks, move to boosted trees, and finally to neural networks.
You can download the data set and workbooks from here.
We will follow these steps:
Load the data, and some of the most required dependencies
Do some minimal preprocessing
Build a couple of straightforward models that will serve as a foundation for further attempts
Spend some time on feature engineering
Apply various methods that will help us deal with unbalanced data
We’ll use the Framingham Heart Study data set from Kaggle for this exercise. It presents a binary classification problem in which we need to predict a value of the variable “TenYearCHD” (zero or one) that shows whether a patient will develop a heart disease.
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport scipy.stats as stimport seaborn as snsimport pandas_profiling%matplotlib inlinedf = pd.read_csv(r'path to dataset')
Let’s make the preliminary data exploration a little bit more convenient. I recently came across an article called “10 Simple hacks to speed up your Data Analysis in Python” written by Parul Pandey and installed a Profiling package.
With just one command, it does some really handy stuff:
# looking at statspandas_profiling.ProfileReport(df)
As you can see, the data is fairly clean: we have missing values here and there but we will handle those soon.
Correlations between features in the chart below don’t say much either:
Two red rectangles around smoking and diastolic/systolic blood pressures are fairly evident: usually smoking and high blood pressure are related
Most of the features’ names give a good understanding of what every variable means but a few of them are less obvious:
“education” can be:
1 — for some high school
2 — for high school of GED
3 — some college or vocational school
4 — college
BPMeds is a binary variable where
0 — means that a patient doesn’t take any blood pressure medications
1 — means the opposite
Let’s examine the target variable manually:
df['TenYearCHD'].value_counts(normalize = True)
And if you’re a visual person:
sns.countplot(x='TenYearCHD',data=df)
It’s not a severely unbalanced set but might skew both linear and non-linear algorithms.
I can think of a few immediate issues here:
missing values and NaNs
Column “Education” may or may not be treated as ordinal
Normalization or standardization of data (usually it belongs to this part but I will do it later, once we start building a model that requires normalized features)
“BPMeds” has some missing values but ~96% of the columns is zero (no blood pressure medications taken). So, it should be fair to fill NaNs with zeroes.
“glucose” and “totChol” are both continuous variables and we will use an average column’s value to fill out empty cells.
“BMI” and “heartRate” seem to work fine with an average, as well.
“cigsPerDay” requires a two-step approach. It has 29 missing values. You may think that using an average is fine. However, “cigsPerDay” is linked to another binary feature “currentSmoker.” So, you could have NaN for a non-smoker and assign some average number of cigarettes to this person. We want to avoid this scenario. Let’s see how that can be done.
df['cigsPerDay'].value_counts(normalize = True).plot(kind="bar")
Most of the people are non-smokers, we don’t want to assign them any cigarettes.
df['cigsPerDay'][df['currentSmoker']==0].isna().sum()
This command returns zero, so it seems that we don’t have NaNs for any non-smokers. But using .mean() is still bad, as it will be skewed towards zero. We want to apply the fillna() command only to smokers:
# creating a boolean array of smokerssmoke = (df['currentSmoker']==1)# applying mean to NaNs in cigsPerDay but using a set of smokers onlydf.loc[smoke,'cigsPerDay'] = df.loc[smoke,'cigsPerDay'].fillna(df.loc[smoke,'cigsPerDay'].mean())
As a result, the average number of cigarettes smoked now is just above 18:
df['cigsPerDay'][df['currentSmoker']==1].mean()
If we decided to go an easier way, without excluding non-smokers, this value would be around nine.
You can be certain that non-smokers are not assigned any values:
df['cigsPerDay'][df['currentSmoker']==0].mean()
“education” is an interesting one. While it’s a categorical variable per se, it’s already encoded for us, and a higher level of education corresponds to a bigger number. A potential problem here is that an “interval” between “college” and “some college or vocational school” likely isn’t the same as between “high school or GED” and “some college or vocational school.” However, the numbers say the opposite: the distance between four and three is exactly the same as between two and three. You can read about this problem by Googling “Ordinal values in logistic regression” or by just going through this paper. Let’s assume for now that the distances are similar. That said, we will just get rid of NaNs in “education” so that a regression can work with the column.
df['education'].value_counts(normalize = True).plot(kind="bar")
It seems fair to substitute NaNs with “1” here.
Making all the changes that are due at once:
# Filling out missing valuesdf['BPMeds'].fillna(0, inplace = True)df['glucose'].fillna(df.glucose.mean(), inplace = True)df['totChol'].fillna(df.totChol.mean(), inplace = True)df['education'].fillna(1, inplace = True)df['BMI'].fillna(df.BMI.mean(), inplace = True)df['heartRate'].fillna(df.heartRate.mean(), inplace = True)
Let’s check that it went through:
df.isna().sum()
And we’re good to go:
We will start with a random forest and will also look at the importance metrics of our features.
Don’t forget to separate features from the target variable:
features = df.iloc[:,:-1]result = df.iloc[:,-1]
The model itself is below.
Dependencies first:
import sklearnfrom sklearn.model_selection import train_test_splitfrom sklearn.linear_model import LogisticRegressionfrom sklearn.metrics import classification_reportfrom sklearn.metrics import confusion_matrixfrom sklearn.metrics import accuracy_scorefrom sklearn.metrics import roc_auc_scorefrom sklearn.preprocessing import StandardScaler
Train/test split:
X_train, X_test, y_train, y_test = train_test_split(features, result, test_size = 0.2, random_state = 14)
Fitting:
rf = RandomForestClassifier()rf.fit(X_train, y_train)# Making predictions on unseen datapredictions_rf = rf.predict(X_test)
Features’ importance (sometimes, for the sake of speed, you want to build a model using fewer features):
# what features are the most important?plt.plot(rf.feature_importances_)plt.xticks(np.arange(X_train.shape[1]), X_train.columns.tolist(), rotation=90)
If you want it as a list:
# View a list of the features and their importance scoreslist(zip(features, rf.feature_importances_))
Assessing the model:
print(classification_report(y_test, predictions_rf))print(confusion_matrix(y_test, predictions_rf))# Under ROC curveprob_rf = rf.predict_proba(X_test)prob_rf = [p[1] for p in prob_rf]print(roc_auc_score(y_test, prob_rf))
Two conclusions here:
The accuracy score of 84.90% just hardly beats a random guessing. Remember that 84.81% of the data is labeled as zero. So we are beating a guess by 0.09%.
Also, the confusion matrix shows a high number of false negatives:
We want to avoid because the model incorrectly states that a patient doesn’t have a problem while in reality he or she does! We will address the problem shortly but for now, let’s rebuild the model using only important features.
from sklearn.feature_selection import SelectFromModelfrom sklearn.ensemble import RandomForestClassifier# Create a selector object that will use the random forest classifier to identify# features that have an importance of more than 0.12sfm = SelectFromModel(clf, threshold=0.12)# Train the selectorsfm.fit(X_train_std, y_train)feat_labels = list(features.columns.values) # creating a list with features' namesfor feature_list_index in sfm.get_support(indices=True):    print(feat_labels[feature_list_index])importances = clf.feature_importances_std = np.std([tree.feature_importances_ for tree in clf.estimators_],             axis=0)indices = np.argsort(importances)[::-1]print("Feature ranking:")for f in range(X_train_std.shape[1]):    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))# Plot the feature importances of the forestplt.figure()plt.title("Feature importances")plt.bar(range(X_train_std.shape[1]), importances[indices],       color="r", yerr=std[indices], align="center")plt.xticks(range(X_train_std.shape[1]), indices)plt.xlim([-1, X_train_std.shape[1]])plt.show()# with only important features. Can check X_important_train.shape[1]X_important_train = sfm.transform(X_train_std)X_important_test = sfm.transform(X_test_std)clf_important = RandomForestClassifier(n_estimators=10000, random_state=0, n_jobs=-1)clf_important.fit(X_important_train, y_train)predictions_y_4 = clf_important.predict(X_important_test)print(classification_report(y_test, predictions_y_4))print(confusion_matrix(y_test, predictions_y_4))accuracy_score(y_test, predictions_y_4)# Under ROC curveprob_y_4 = clf_important.predict_proba(X_important_test)prob_y_4 = [p[1] for p in prob_y_4]print(roc_auc_score(y_test, prob_y_4))
I selected features with the importance more than 0.12 and rebuilt the random forest using only these columns. The results came in pretty similar but we were able to save some computational power.
Regressions need standardized features:
scaler = StandardScaler()X_train_std = scaler.fit_transform(X_train)X_test_std = scaler.fit_transform(X_test)
And the model:
logmodel = LogisticRegression(solver='liblinear')logmodel.fit(X_train_std, y_train)predictions_y_2 = logmodel.predict(X_test_std)
And assessing results:
print(classification_report(y_test, predictions_y_2))print(confusion_matrix(y_test, predictions_y_2))# Under ROC curveprob_y_2 = logmodel.predict_proba(X_test_std)prob_y_2 = [p[1] for p in prob_y_2]print(roc_auc_score(y_test, prob_y_2))
Results are very similar to random forest’s:
It did slightly better on false negatives. But it’s still not enough.
The LogisticRegression has a parameter class_weight that should help more accurate. Let’s see:
logmodel = LogisticRegression(solver='liblinear', class_weight='balanced')logmodel.fit(X_train_std, y_train)predictions_y_3 = logmodel.predict(X_test_std)print(classification_report(y_test, predictions_y_3))print(confusion_matrix(y_test, predictions_y_3))accuracy_score(y_test, predictions_y_3)# Under ROC curveprob_y_3 = logmodel.predict_proba(X_test_std)prob_y_3 = [p[1] for p in prob_y_3]print(roc_auc_score(y_test, prob_y_3))
It’s not really helpful: we decrease the number of Type II errors at some expense of the overall accuracy. It seems we need to move on to something different.
We can also implement class_weight manually by passing a breakdown of zero and one values that you can quickly get using value_counts:
df['TenYearCHD'].value_counts(normalize = True)weights = {0 : '0.848113', 1 : '0.151887'}logmodel_auto = LogisticRegression(class_weight = weights, solver = 'liblinear')logmodel_auto.fit(X_train_std, y_train)predictions_std_auto = logmodel_auto.predict(X_test_std)print(classification_report(y_test, predictions_std_auto))print(confusion_matrix(y_test, predictions_std_auto))accuracy_score(y_test, predictions_std_auto)# Under ROC curveprob_y_4 = logmodel.predict_proba(X_test_std)prob_y_4 = [p[1] for p in prob_y_4]print(roc_auc_score(y_test, prob_y_4))
Good job on Type I errors, actually:
You can try different combinations. By trial and error (or with the help of GridSearch), you can find one that serves your purpose, for example:
weights = {0 : '0.09042', 1 : '0.90958'}
If you decide to go with GridSearch to find proper weights, this is how it can be implemented:
from sklearn.model_selection import GridSearchCVweights = np.linspace(0.03, 0.97, 55)scaler = StandardScaler()features_std = scaler.fit_transform(features)gsc = GridSearchCV(    estimator=LogisticRegression(solver='liblinear'),    param_grid={        'class_weight': [{0: x, 1: 1.0-x} for x in weights]    },    scoring='roc_auc',    cv=3)grid_result = gsc.fit(features_std, result)
In the code above, we test different combinations of zeroes and ones for the variable TenYearCHD going from a 3-97% breakdown to a 97-3% breakdown. GridSearchCV can evaluate different estimators, and they are chosen and controlled in the field scoring. Usually, you would use something like accuracy for classification and R-squared for regression, but in our case accuracy won’t be very informative, so I decided to go with the area under the AUC curve. All available scoring parameters are described here.
Then we can just print out the best parameters and pass it back to the model:
print("Best parameters : %s" % grid_result.best_params_)# passing weights found aboverf_w = RandomForestClassifier(class_weight = {0:0.882962962962963, 1:0.11703703703703705})rf_w.fit(X_train, y_train)print(classification_report(y_test, predictions_rf_w))print(confusion_matrix(y_test, predictions_rf_w))accuracy_score(y_test, predictions_rf_w)
Or to the logistic regression:
weights = {0 : '0.882962962962963', 1 : '0.11703703703703705'}logmodel_auto_gridsearch = LogisticRegression(class_weight = weights, solver = 'liblinear')logmodel_auto_gridsearch.fit(X_train_std, y_train)predictions_std_auto_gridsearch = logmodel_auto_gridsearch.predict(X_test_std)print(classification_report(y_test, predictions_std_auto_gridsearch))print(confusion_matrix(y_test, predictions_std_auto_gridsearch))accuracy_score(y_test, predictions_std_auto_gridsearch)# Under ROC curveprob_y_3_gridsearch = logmodel_auto_gridsearch.predict_proba(X_test_std)prob_y_3_gridsearch= [p[1] for p in prob_y_3_gridsearch]print(roc_auc_score(y_test, prob_y_3_gridsearch))
One of the most popular methods is up(down)sampling. The problem with our data set, as you may remember, is that algorithms see drastically more negative (zero) cases than positive during the training stage, and it makes the models less accurate. The next logical thing to do is to train a model on the same number of positive (1) and negative (0) cases.
from sklearn.utils import resampledf_minority = df[df.TenYearCHD==1]df_majority = df[df.TenYearCHD==0]df['TenYearCHD'].value_counts()
In the code above, we separate the existing dataset into two: observations that have developed a heart disease and the rest.
# sample with replacement to match majority class and get #reproducible resultsdf_minority_upsampled = resample(df_minority,                                  replace=True,                                      n_samples=3596,                                     random_state=123) # Display new class countsdf_upsampled.TenYearCHD.value_counts()
As you see now, df_upsampled has the same number of zero and one observations.
Now we will normalize it and throw into a model again:
# Train/test, normalize the new data setfeatures_upsampled = df_upsampled.iloc[:,:-1]result_upsampled = df_upsampled.iloc[:,-1]X_train_upsampled, X_test_upsampled, y_train_upsampled, y_test_upsampled = train_test_split(features_upsampled, result_upsampled, test_size = 0.2, random_state = 14)X_train_std_upsampled = scaler.fit_transform(X_train_upsampled)X_test_std_upsampled = scaler.fit_transform(X_test_upsampled)# new log model for upsampled datalogmodel_upsampled = LogisticRegression(solver='liblinear')logmodel_upsampled.fit(X_train_std_upsampled, y_train_upsampled)predictions_y_2_upsampled = logmodel_upsampled.predict(X_test_std_upsampled)print(classification_report(y_test_upsampled, predictions_y_2_upsampled))print(confusion_matrix(y_test_upsampled, predictions_y_2_upsampled))accuracy_score(y_test_upsampled, predictions_y_2_upsampled)# Under ROC curveprob_y_2_upsampled = logmodel_upsampled.predict_proba(X_test_std_upsampled)prob_y_2_upsampled = [p[1] for p in prob_y_2_upsampled]print(roc_auc_score(y_test_upsampled, prob_y_2_upsampled))
In our case, results have deteriorated. Why so? We will explore it next time. Usually it helps, though.
We have already mentioned that Errors II are more critical in this case than Errors I. Another way to account for that is to move the threshold (that is now set up to 0.5).
logmodel_lowering = LogisticRegression(solver='liblinear')logmodel_lowering.fit(X_train_std, y_train)from sklearn.preprocessing import binarizefor i in range(1,7):    cm2=0    predictions_y_2_lowering = logmodel_lowering.predict_proba(X_test_std)    y_pred2_lowering=binarize(predictions_y_2_lowering,i/10)[:,1]    cm2=confusion_matrix(y_test,y_pred2_lowering)    print ('With',i/10,'threshold the Confusion Matrix is ','\n',cm2,'\n',            'with',cm2[0,0]+cm2[1,1],'correct predictions and',cm2[1,0],'Type II errors( False Negatives)','\n\n',          'Sensitivity: ',cm2[1,1]/(float(cm2[1,1]+cm2[1,0])),'Specificity: ',cm2[0,0]/(float(cm2[0,0]+cm2[0,1])),'\n\n\n')
We wrote a loop here to go from a threshold of 10% to the threshold of 70% and then show the results. I’m fitting the basic logistic regression model here for the sake of this exercise.
So far we have made only limited progress. It’s probably time to turn to XGBoost — a weapon of choice for the most Kaggle competitions.
import xgboost as xgbfrom sklearn.metrics import mean_squared_errorxg_reg = xgb.XGBRegressor(objective ='binary:logistic', colsample_bytree = 0.3, learning_rate = 0.05,max_depth = 9, alpha = 10, n_estimators = 20)eval_set = [(X_test_std, y_test)]xg_reg.fit(X_train_std, y_train, eval_metric="error", eval_set = eval_set, verbose = True)rmse = np.sqrt(mean_squared_error(y_test, prediction_y_5))print("RMSE: %f" % (rmse))
Unlike other methods, XGBoost can report and evaluate the performance of the test set during the fit stage. I’ve created and eval_set, passed it to the fit method and set up verbose = True to see verbosity in the real time.
Out of the box, it returns a root mean square error (RMSE) of 0.3678. Let’s work on lowering it. To do so, we need to tweak some of the numerous parameters that are passed in the algorithm. You really need to read about them in the official documentation because it’s the key to achieving the superb performance of your model. Today I will focus at three of the available parameters. And again, GridSearchCV will do it for us.
We are going to test various inputs of these parameters:
n_estimators (defines the size of the forest to be trained)
max_depth (maximum tree depth for base learners)
learning_rate (well, it’s a learning rate)
n_estimators = [10, 20, 30, 40, 50, 60]max_depth = [2, 4, 5, 6, 7, 8]learning_rate = [0.0001, 0.001, 0.01, 0.1, 0.2, 0.3]param_grid = dict(max_depth = max_depth, n_estimators = n_estimators, learning_rate=learning_rate)kfold = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 10)grid_search_xg = GridSearchCV(xg_reg, param_grid, scoring = 'roc_auc', n_jobs = -1, cv=kfold, verbose = 1)result_gcv_xgb = grid_search_xg.fit(X_train_std, y_train)
We do cross-validation here using StratifiedKFold and GridSearchCV to iterate over parameters. The more parameters you want to test, the more permutations your computer has run, and the more time it will take. Be cautious if you are doing it on your laptop.
Once the process is completed, let’s see what the best parameters are:
print("Best: %f using %s" % (result_gcv_xgb.best_score_, result_gcv_xgb.best_params_))means = result_gcv_xgb.cv_results_['mean_test_score']stds = result_gcv_xgb.cv_results_['std_test_score']params = result_gcv_xgb.cv_results_['params']
Then you can rerun the model using what you’ve gotten above:
# rebuild using best paramsxg_reg = xgb.XGBRegressor(objective ='binary:logistic', colsample_bytree = 0.3, learning_rate = 0.1,max_depth = 2, alpha = 10, n_estimators = 50)eval_set = [(X_test_std, y_test)]xg_reg.fit(X_train_std, y_train, eval_metric="error", eval_set = eval_set, verbose = False)prediction_y_5 = xg_reg.predict(X_test_std)rmse = np.sqrt(mean_squared_error(y_test, prediction_y_5))print("RMSE: %f" % (rmse))
We can see an improvement: RMSE went down to 0.3384.
XGBoost returns probabilities, not actual predictions. We will need actual predictions for building a confusion matrix, though.
prediction_y_5_01 = prediction_y_5prediction_y_5_01[prediction_y_5 > 0.5] = 1prediction_y_5_01[prediction_y_5 <= 0.5] = 0print(classification_report(y_test, prediction_y_5_01))print(confusion_matrix(y_test, prediction_y_5_01))accuracy_score(y_test, prediction_y_5_01)
We have tried using some traditional and more specific methods to build an accurate prediction model. Depending on the nature of your data, it might be already enough for improving model’s performance. In the second part of the article, we will continue our journey by building a learning curve, an ensemble of a few models, and finally rerunning the analysis using Keras.