Classification in Imbalanced Data Sets. | by Ada kibet | Towards Data Science

Classification is a type of supervised learning in Machine Learning that deals with categorizing data into classes. Supervised learning implies models that take input with their matched output to train a model which can later make useful predictions on a new set of data with no output. Some examples of classification problems include: spam detection in mails, subscription analysis, hand written digit recognition, survival predictions et.c. They all involve the use of classifiers that utilize the train data to understand how the input variables relate to the output (target) variable.
Class imbalance refers to a problem in classification where the distribution of the classes are skewed. This can range from a slight to an extreme imbalance.
This is a problem because most classification algorithms will have a low prediction accuracy towards the minority class because they run with the assumption that there’s balance between the classes.
An example of class imbalance is in credit card fraud detection. The classes, in this case, are fraud and not fraud. Most of the transactions are not fraud, thus the fraud classes are the minority class. Having a low accuracy in the minority class predictions is problematic because it is the most important class.
This blog covers the steps involved in tackling a classification problem in imbalanced dataset. The Github repository containing all the code is available here.
Data used is from the UCI Machine Learning Repository. The data is related with marketing campaigns of a Portuguese banking institution. The classification goal is to predict if the client will subscribe to a term deposit (variable y).
An effective model can help increase campaign efficiency as more efforts can be directed towards customers with high subscription chances.
Sample of the data:
Visualizing the target variable(y) in order to observe the class imbalance:
The circles’ sizes represents the value counts of each class. Clearly there’s extreme class imbalance. This will be taken care of in the preprocessing section.
Preprocessing involves the following steps:
Missing values need to be handled because they can lead to wrong predictions and can also cause a high bias for any given model being used.
The categorical features will be imputed with the column mode, the discrete numerical features with the column median and the continuous numerical features with the column mean.
Outliers are a problem for many machine learning algorithms because they can cause missing of important insights or contort real results which eventually results to less accurate models.
The outliers will be clipped with the 10th and 90th percentiles.
Generating new features from already existing features adds new information to be accessible during the model training and therefore increasing model accuracy.
Standardizing numerical features with a StandardScaler to remove the differences brought about by different units of measurement.
Most Machine Learning and Neural Nets algorithms require numerical inputs, thus to make use of the categorical features, we have to remap them.
One hot encoding technique is applied. It takes a column which has categorical data then splits the column into multiple columns. The entries are replaced by zeros and ones, depending which column has what value.
Resampling involves creating a new version of our imbalanced dataset.
There are 2 main approaches for resampling:
Over sampling: Randomly duplicating entries in the minority class. Appropriate for small datasets.
Under sampling: Randomly deleting entries from the majority class. Appropriate for large datasets.
Our dataset has 41188 rows and 21 columns thus safe to use oversampling.
The Preprocessing class:
# calling the class and its methodsd = DataPrep()path = '/content/bank-additional-full.csv'data = d.read_data(path)data = d.treat_null(data)data = d.outlier_correcter(data)data = d.generate_features(data)data = d.scaler(data)print('After scaling:', data.shape)data = d.encoder(data)data = d.over_sample(data)data.head()
The output:
Notice the changes in the categorical and numerical columns.
Straight off from the preprocessing to making predictions, we begin by training and evaluating our models using the train and validation data. But first, we have to separate the target and predictor variables then split them into train, validation and test sets. The test set is not provided separately thus we retrieve it from our dataset.
# split the data to have the predictor and predicted variablesx = data.drop(['y'], axis = 1)y = data[['y']]# Encode labels in target df.encoder = LabelEncoder()y = encoder.fit_transform(y)# get the setsx_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.10, random_state = 42)x_train, x_val, y_train, y_val = train_test_split(x_train ,y_train, test_size = 0.20, random_state = 42)
Algorithms to be explored:
XGBoost:
XGBoost classifier is tree-based ensemble learning algorithm that’s an implementation of gradient boosting machines.
It is optimized for speed and performance.
* Multilayer Perceptron:
A multilayer perceptron (MLP) is a class of feed-forward artificial neural network. It consists of at least three layers of nodes: an input layer, a hidden layer and an output layer.
Its ability to distinguish data that is not linearly separable is why we explore it.
* Logistic Regression:
Logistic regression is a simple and yet very effective classification algorithm that uses log odds ratio to predict group membership.
Its simplicity and use of log odds ratio in place for probabilities is why we explore it.
Cross-validation is a resampling procedure used to evaluate machine learning models on a limited data sample.
Techniques used:
K-fold: K-Fold CV is where a given data set is split into n number of folds where each fold is used as a testing set at some point and the rest as the training set.
Stratified K-fold: StratifiedKFold shuffles the data, then splits it into n number of folds then the folds are each used as a testing set. Stratification keeps the balance between targets of the dataset (each stratified fold keeps the same ratio of the target classes). This strategy is best in cases of imbalanced data.
The scoring metric by default in both techniques is the accuracy score (the number of correct predictions made divided by the total number of predictions made).
Implementation:
Now we observe our models performance on the train data using these techniques.
# using grid search to find optimal parametersregressor = LogisticRegression()grid_values = {'penalty': ['l1','l2'], 'C': [0.001,0.01,0.1,1,10,100,1000]}model = GridSearchCV(regressor, param_grid=grid_values)model.fit(x_train,y_train)print(model.best_score_)print(model.best_params_)
# using the optimal parametes printed outregressor = LogisticRegression(C = 1000, penalty= 'l2')regressor.fit(x_train, y_train)# using kfoldsprint('Logistic Regression mean accuracy score using kfold:', overall_score(regressor, x_train))# stratified KFoldprint('Logistic Regression mean accuracy score using Stratifiedkfold :', overall__stratified_score(regressor, x_train))
Output:
Logistic Regression mean accuracy score using kfold: 0.742437522099093Logistic Regression mean accuracy score using Stratifiedkfold : 0.7420879248958712
Repeating the same procedure for the XGBoost and the MLP.
xgb = XGBClassifier(silent = True,max_depth = 6, n_estimators = 200)xgb.fit(x_train, y_train)# using kfoldsprint('xgb mean score on the original dataset (kfold):', overall_score(xgb, x_train))# stratified KFoldprint('xgb mean score on the original dataset (stratified kfold):', overall__stratified_score(xgb, x_train))
mlp = MLPClassifier() # working with default parametersmlp.fit(x_train, y_train)# using kfoldsprint('mlp mean score on the original dataset (kfold):', overall_score(mlp, x_train))# stratified KFoldprint('mlp mean score on the original dataset (stratified kfold):', overall__stratified_score(mlp, x_train))
In all the 3 models, K-fold yields the highest accuracy although its variation with Stratified is so slight. Like mentioned earlier, Stratified is best for imbalanced data but we already resampled our data during preprocessing rendering it not so useful.
We can explore other accuracy metrics through our validation set.
ROC — ROC curve is a plot of the true positive rate (y-axis) against the false positive rate (x-axis) for a number of different candidate threshold values between 0.0 and 1.0.
PRECISION AND RECALL — Precision is the number of correctly identified positive results divided by the number of all positive results, including those not identified correctly and recall is the number of correctly identified positive results divided by the number of all samples that should have been identified as positive.
F1 SCORE — F1 score is a measure of a test’s accuracy. It is calculated from the precision and recall of the test.
ROC curves should be used when there are roughly equal numbers of observations for each class. Precision-Recall curves should be used when there is moderate to large class imbalance.
Since we resampled our data to be balanced, ROC is our best take.
ROC plots: Implementation :
The plots are implemented on other models: RandomForest, CatBoost and LGBM, they can be edited to fit whichever model. (Code available in the github link shared).
from sklearn.datasets import make_classificationfrom sklearn.metrics import roc_curvefrom sklearn.metrics import roc_auc_score#predict probabilitiesns_probs = [0 for _ in range(len(x_val))]  # no skill classifierf_prob = forest.predict_proba(x_val)c_prob = cat.predict_proba(x_val)l_prob = lgbm.predict_proba(x_val)# keep probabilities for the positive outcome onlyf_prob = f_prob[:, 1]c_prob = c_prob[:, 1]l_prob = l_prob[:, 1]# calculate scores then print themf_auc = roc_auc_score(y_val, f_prob)c_auc = roc_auc_score(y_val, c_prob)l_auc = roc_auc_score(y_val, l_prob)ns_auc = roc_auc_score(y_val, ns_probs)print('RandomForest:', f_auc)print('CatBoost: ', c_auc)print('LGBM:', l_auc)# calculate roc curvesf_fpr, f_tpr, _ = roc_curve(y_val, f_prob)c_fpr, c_tpr, _ = roc_curve(y_val, c_prob)l_fpr, l_tpr, _ = roc_curve(y_val, l_prob)ns_fpr, ns_tpr, _ = roc_curve(y_val, ns_probs)# plot the roc curve for the modelplt.figure(figsize = (12,7))plt.plot(f_fpr, f_tpr, marker='.', label='random forest')plt.plot(l_fpr, l_tpr, marker='.', label='lgbm')plt.plot(c_fpr, c_tpr, marker='.', label='catboost')plt.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')plt.legend()plt.title('ROC curves for different models')plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')
Output:
The bigger the area under the curve, the higher the accuracy.
A no-skill classifier is one that cannot discriminate between the classes and would predict a random class or a constant class in all cases. (Our benchmark).
Using MLP, our best performer from the 3 initial models.
# mlppred = mlp.predict(x_test)pred_df = pd.DataFrame(pred)pred_df.columns = ['y']pred_df.to_csv('pred_df.csv') # export to a csv file
We could go on and on with our classification problem exploring distinct techniques like Dimensionality reduction solely in an attempt to achieve a better performing model. But for now, we know that having imbalanced data doesn’t hinder one from performing predictions, it simply beckons one to use the appropriate techniques to evade poor forecasts on the minority class.
PS: Respect to all my fellow learners and tutors at 10academy.org for their endless support.
How to Use ROC Curves and Precision-Recall Curves for Classification in Python.
How to Use ROC Curves and Precision-Recall Curves for Classification in Python.
2. Understanding AUC — ROC Curve.
3. XGBoost Algorithm: Long May She Reign!
4. Dealing with Imbalanced Data.
5. XGBoost Documentation.
6. Imbalanced Data : How to handle Imbalanced Classification Problems.
7. Why Is Imbalanced Classification Difficult?