A Simple Guide to Handle Imbalanced Classification: A step-by-step approach
By: Neha Mangal
Tags: Data Science, Imbalanced Class, Classification

Imbalanced class classification is basically when there is unequal distribution in the target classes. In this article, I am outlining a high-level approach to tackle these imbalance problem when we want to identify the minority class.

Select an evaluation metric to best represent the desired solution Choose suitable algorithms to work on imbalance data Tune hyperparameters to improve performance

If you are not interested in reading this whole article, you can simply refer my notebook containing most of these methods!!

Okay, Let’s get into details:

SELECT AN EVALUATION METRIC: This is the most important step in solving a problem, especially in imbalanced classification, before even thinking or trying any algorithm. The metric is the measuring stick by which all models are evaluated and compared. The choice of the wrong metric can mean choosing the wrong algorithm. Remember, in a real world imbalanced classification problems, classification accuracy is not the right metric in most of the cases.

Problems where we want to predict Class Probabilities :

In case probabilities are to be used directly, a good metric might be the Brier Score and the Brier Skill score. Alternatively, we may want to map the prediction probabilities to crisp class labels via a user-selected threshold. For this, we can choose a measure to summarise the model performance across all the possible thresholds.

Recall that for binary imbalanced classification tasks, the majority class is normal, called the “negative class“, and the minority class is the exception, called the “positive class“.

If positive class is most important (for example HIV identification), then the precision-recall curve and area under curve (PRC, AUC) can be used. This will optimize both precision and recall across all thresholds.

Alternately, if both classes are equally important(Fraud detection), the ROC Curve and area under curve (ROC, AUC) can be used. This will maximize the true positive rate and minimize the false positive rate .

: In case probabilities are to be used directly, a good metric might be the Brier Score and the Brier Skill score. Alternatively, we may want to map the prediction probabilities to crisp class labels via a user-selected threshold. For this, we can choose a measure to summarise the model performance across all the possible thresholds. Recall that for binary imbalanced classification tasks, the majority class is normal, called the “negative class“, and the minority class is the exception, called the “positive class“. If positive class is most important (for example HIV identification), then the precision-recall curve and area under curve (PRC, AUC) can be used. Alternately, if both classes are equally important(Fraud detection), the ROC Curve and area under curve (ROC, AUC) can be used. This will . Problems where we want to predict Class Labels:

a. Classification accuracy : only when majority class is less than 80%

b. G-Mean metric: in severely skewed data

c. F1, F2, F0.5 measures: depending on which class is more costly(important) : F1: if both are equally important, F2: if false negatives are more costly; F0.5: if false positives are more costly.

Here is the summary of how you can choose a right eval metric for your problem.

Summary of selecting a metric for Imbalanced classification. Image Source: MachineLearningMastery

For scoring parameters details, read: scikit-learn scoring parameters

2. SELECT A SUITABLE ALGORITHM:

Comparison of different Classifier algorithms on Imbalanced data(data source: DataMasked); y_axis: ROC_AUC_score

Remember, the basic idea is same here. We use cross-validation scores (using the chosen evaluation metric) to select from our candidate standard ML algorithms. However, in imbalanced classification, we need to do a little bit more work on these standard ML algorithms and/or appropriate sampling since we may not get the desired goal(good recall etc.) due to the skewed data. I am outlining the most often used techniques for creating imbalanced candidate algorithms:

(Algorithm centric) Cost-sensitive algorithms: Most often, this approach alone gives the desired results through focusing on minimising the misclassification cost. The main idea behind this approach is: since, the default classifier algorithm gives equal weight to the cost of correct and incorrect classification; so, custom specifying the costs for minority and majority class should tackle the predictions.

These include: changing prediction probabilities threshold (tackles the class costs after the training) and class reweighing (tackles the class costs during the training itself).

For example: In a fraud detection problem(0.7% minority class ratio), reducing the prediction probabilities from a default 0.5 to 0.2(that means, if prediction prob < 0.2, clf will assign it as the minority class) would give better results.

Note, for finding he optimum thresholds, we can choose a no of methods, a few of them are max(Gmeans), max(fpr*(1-tpr)) etc. for class weights, we can choose cross validation methods to choose optimum. different algorithms such as LR, GBM, XGBoost have different methods of setting the class weights. You should read first about the setting these weight parameters before assigning any random values as it can slow down the training/CV process.

Most often, this approach alone gives the desired results through focusing on minimising the misclassification cost. The main idea behind this approach is: since, the default classifier algorithm gives equal weight to the cost of correct and incorrect classification; so, custom specifying the costs for minority and majority class should tackle the predictions. These include: (tackles the class costs after the training) and (tackles the class costs during the training itself). For example: In a fraud detection problem(0.7% minority class ratio), reducing the prediction probabilities from a default 0.5 to 0.2(that means, if prediction prob < 0.2, clf will assign it as the minority class) would give better results. Note, for finding he optimum thresholds, we can choose a no of methods, a few of them are max(Gmeans), max(fpr*(1-tpr)) etc. for class weights, we can choose cross validation methods to choose optimum. different algorithms such as LR, GBM, XGBoost have different methods of setting the class weights. You should read first about the setting these weight parameters before assigning any random values as it can slow down the training/CV process. (Data centric) Resampling methods: Sometimes reworking the algorithms as cost-sensitive alone does not provide with the desired results. Also, in order to do sampling for training, we sometimes need these resampling methods to make the data more balanced for the minority class. There are a bunch of techniques used for this:

— Data Oversampling : Data oversampling involves duplicating examples of the minority class or synthesising new examples from the minority class from existing examples. These include: Random Oversampling, SMOTE (with different variations such as borderline SMOTE, KNN SMOTE, SVM SMOTE etc.) , ADASYN etc.

— Data Undersampling : Undersampling involves deleting examples from the majority class, such as randomly or using an algorithm to carefully choose which examples to delete. These include: Near Miss, Tomek links, Edited Nearest Neighbors, Random Undersampling etc

— Combination: popular examples: SMOTE and Tomek Links, SMOTE and Random Undersampling

Caution : Oversampling the minority can lead to model overfitting, since it will introduce duplicate instances, drawing from a pool of instances that is already small. Similarly, undersampling the majority can end up leaving out important instances that provide important differences between the two classes. Also, resampling methods use the nearest neighbor algorithm internally, so it is best to normalise and scale the data before applying these methods. Also, these methods do not work best with ensemble of trees so, avoid using them with these models.

Sometimes reworking the algorithms as cost-sensitive alone does not provide with the desired results. Also, in order to do sampling for training, we sometimes need these resampling methods to make the data more balanced for the minority class. There are a bunch of techniques used for this: — : Data oversampling involves duplicating examples of the minority class or synthesising new examples from the minority class from existing examples. These include: Random Oversampling, SMOTE (with different variations such as borderline SMOTE, KNN SMOTE, SVM SMOTE etc.) , ADASYN etc. — : Undersampling involves deleting examples from the majority class, such as randomly or using an algorithm to carefully choose which examples to delete. These include: Near Miss, Tomek links, Edited Nearest Neighbors, Random Undersampling etc — popular examples: SMOTE and Tomek Links, SMOTE and Random Undersampling : Oversampling the minority can lead to model overfitting, since it will introduce duplicate instances, drawing from a pool of instances that is already small. Similarly, undersampling the majority can end up leaving out important instances that provide important differences between the two classes. Also, There are a bunch of other approaches such as: outlier/anomaly detection(one-class classification) for dataset where there is a severe skewness; and probability tuning algorithms(where prediction probabilities can be calibrated/tuned to evaluate the model)

A summary of selecting a suitable algorithm. Note that the order can be flexible as opposed to shown by sequential downward arrows in different algorithms. Image Source: MachineLearningMastery

Note: While these approaches outline the methods to work the models towards the desired goal, one should always apply these methods to a number of candidate algorithms (for example: LR, BoostedTress, SVM etc.) and select the best performing algorithm by using the cross-validation methods(for example: RepeatedStratifiedKFold usually works best with the imbalanced data)with the score parameter set as the evaluation metric. Of course, for comparison you must have a baseline model (which can be achieved by a simple dummy classifier

3. Hyperparameter Tuning: Most common techniques to tune the model hyperparameters are:

Grid Search

Random Search

Once, we have some idea of what algorithms may work best on our dataset, we can choose the top 5–10 algorithms and use the above techniques to tune their respective hyperparameters.

There may be some standard machine learning algorithms that perform well, but do not perform as well when used with data sampling or probability calibration. These algorithms could be tuned in concert with their imbalanced-classification augmentations to see if better performance can be achieved.

Additionally, there may be imbalanced-classification algorithms, such as a data sampling method that results in a dramatic lift in performance for one or more algorithms. These algorithms themselves may provide an interesting basis for further tuning to see if additional lifts in performance can be achieved.

Conclusion/Rule of Thumb

whenever using a machine learning algorithm, evaluation metrics for the model have to be chosen cautiously: we must use the metrics that gives us the best overview of how well our model is doing with regards to our goals.

resampling methods can be used but have to be thought carefully: they should not be used as stand alone solutions but have to be coupled with a rework of the problem to serve a specific goal

methods can be used but have to be thought carefully: reworking the problem itself is often the best way to tackle an imbalanced classes problem: the classifier and the decision rule have to be set with respect to a well chosen goal that can be, for example, optimizing the cost

I hope, this article helps in defining a high level structure to tackle an imbalanced classification problems. If you have suggestions/doubts, let me know in the comments!!