Practical tips for class imbalance in binary classification | by Zichen Wang | Towards Data Science

Binary classification problem is arguably one of the simplest and most straightforward problems in Machine Learning. Usually we want to learn a model trying to predict whether some instance belongs to a class or not. It has many practical applications ranging from email spam detection to medical testing (determine if a patient has a certain disease or not).
Slightly more formally, the goal of binary classification is to learn a function f(x) that map x (a vector of features for an instance/example) to a predicted binary outcome ŷ (0 or 1). Most classification algorithms, such as logistic regression, Naive Bayes and decision trees, output a probability for an instance belonging to the positive class: Pr(y=1|x).
Class imbalance is the fact that the classes are not represented equally in a classification problem, which is quite common in practice. For instance, fraud detection, prediction of rare adverse drug reactions and prediction gene families (e.g. Kinase, GPCR). Failure to account for the class imbalance often causes inaccurate and decreased predictive performance of many classification algorithms. In this post, I will introduce a couple of practical tips on how to combat class imbalance in binary classification, most of which can be easily adapted to multi-class scenarios.
First and foremost, you want to stratify your data for training and validation. Stratification is the technique to allocate the samples evenly based on sample classes so that training set and validation set have similar ratio of classes.
It is essential to ensure your training and validation sets share approximately the same ratio of examples from each class, so that you can achieve consistent predictive performance scores in both sets. Some classification algorithms are also extremely sensitive to the class ratio of the data that they are trained on. For instance, naive Bayes classifier makes prediction using the class prior it learned from the ratio of classes in the training data. Large discrepancy of class ratios between training and validation sets leads to the inaccurate estimation of class prior which decrease the predictive performance of naive Bayes classifier. Similarly, k-Nearest Neighbors also suffers from this problem since it simply remembers all the data points used for training, which would almost certainly introduce bias stemming from the class imbalance during inference phase as it search for k-nearest neighbors from the training data.
scikit-learn has an implementation for stratification StratifiedKFoldto put that into codes:
We can then compare the scores from stratified and random cross-validations (CV) and it usually makes a difference for classifiers that are sensitive to class imbalance:
There are many metrics for evaluating how good a binary classifier is doing in predicting the class labels for instances/examples. Below are some caveats and suggestions for choosing and interpreting the appropriate metrics.
Accuracy can be misleading. Since accuracy is simple the ratio of correctly predicted instances over all instances used for evaluation, it is possible to get a decent accuracy while having mostly incorrect predictions for the minority class.
Confusion matrix helps break down the predictive performances on different classes.
Bipartition-based metrics (Precision, Recall, F1-score and etc.) by defaults compute the averaged scores from both classes. Perhaps not obvious to many, one can actually get the breakdowns of those scores for each class. The classification_report function is really useful here in this case:
Precision-Recall (PR) curve is recommended over ROC for imbalanced data. As the figure shown below, ROC curves can overestimate the performance of classifier on imbalanced data whereas PR curves reveal the drop in precision at the same level of recall for the imbalanced data. This article has discussed how to choose the appropriate classification metrics in greater details.
Now that we know how to set up the evaluation scheme and what metrics to choose for classification problems with imbalanced data, we can go ahead and apply some techniques to account for class imbalance. The most straightforward technique is to balance the data by resampling:
Down-sampling (Under sampling) the majority class
Up-sampling (Over sampling) the minority class
Many more advanced sampling techniques, such as Synthetic Minority Over-sampling Technique (SMOTE) (Chawla et al., 2012), are implemented in the imbalanced-learn library
Resampling techniques can be coupled with Bagging ensemble models such as Random Forest. Bagging stands for bootstrap-aggregation, they work by bootstrapping the data when building individual estimators (usually decision trees), then aggregate the predictions across individual estimators. Since bootstrapping (random sampling with replacement) is essentially one type of resampling technique, we can replace the bootstrapping with other resampling techniques such as down-sampling or up-sampling to balance the data.
Below are some core Python codes implementing Random Forest with up-/down- sampling. Full codes implementing resampling forest can be found here.
Next, we can compare the performance of Random Forest with different resampling techniques. The figure indicates both up-sampling and down-sampling improve the predictive performance of Random Forest on imbalanced data.
Without resampling the data, one can also make the classifier aware of the imbalanced data by incorporating the weights of the classes into the cost function (aka objective function). Intuitively, we want to give higher weight to minority class and lower weight to majority class. scikit-learn has a convenient utility function to calculate the weights based on class frequencies:
from sklearn.utils.class_weight import compute_class_weightclass_weights = compute_class_weight('balanced', np.unique(y), y)
Cross entropy is a common choice for cost function for many binary classification algorithms such as logistic regression. Cross entropy is defined as:
CrossEntropy = −ylog(p) − (1−y)log(1−p)
, where y is the class binary indicator (0 or 1) and p is predicted probability for instance belonging to class 1. To incorporate the weights of two classes (0 and 1) into the cross entropy, one can define a weighted cross entropy:
, in which w_0 and w_1 are the weights for class 1 and 0, respectively. It is possible to implement class weights in Tensorflow using tf.nn.weighted_cross_entropy_with_logits. In Keras, class_weight can be passed into the fit methods of models as a parameters when training. I will implement examples for cost-sensitive classifiers in Tensorflow and Keras in the future.
Sensitive learning can also be achieved by setting the class_weight parameter in many scikit-learn classifiers such as support vector machine (SVM) and Random Forest.
In Random Forest:
class_weight='balanced': uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input dataclass_weight='balanced_subsample': is the same as “balanced” except that weights are computed based on the bootstrap sample for every tree grown.
class_weight='balanced': uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data
class_weight='balanced_subsample': is the same as “balanced” except that weights are computed based on the bootstrap sample for every tree grown.
Some classification models have built-in approaches combatting class imbalance. For instance, Gradient Boosting Machines (GBM) deals with class imbalance by constructing successive training sets based on incorrectly classified examples. It usually outperforms Random Forest on imbalanced dataset:
I hope these tips are helpful for improving your classification model on imbalanced datasets. A Jupyter notebook version of this tutorial is available on my GitHub:
github.com
Thanks for reading!