How to Predict Imbalanced Classes in Python | Towards Data Science

By a curious twist of fate, the amount of time I generally spend on classification problems is much more than that on regression problems. And as luck would have it, I get to work with imbalanced datasets more often than not. Perhaps this is because my day job involves prediction and analysis of credit risk, probability of default, and other fraud risks!
It is not common for one to come across payment defaults, payment delays, and fraud events on a regular basis in the usual scheme of things. This is to say that the proportion of such events, when compared to payments that are made on time or normal non-fraudulent activities, is usually very low.
This is a very simple illustration of an imbalanced dataset whereby one set (or class) of activity (e.g., payment default) forms a very low proportion of the total number of activities (e.g., defaults and normal payments) in the sample.
Now let’s see why we need to be careful when confronted with, and how to deal with, imbalanced classification problems. The rest of the post is structured as follows:
IntroductionModel Evaluation MetricsCost-Sensitive LearningSampling Methods
Introduction
Model Evaluation Metrics
Cost-Sensitive Learning
Sampling Methods
When working with an imbalanced classification problem, we are usually more concerned with correctly predicting the minority class (also called the positive class), e.g., fraud event, payment default, spam, etc. In other words, it is more critical for us to reduce the number of false negatives (e.g., classifying a fraudulent transaction as non-fraudulent) than false positives (e.g., classifying a normal email as spam).
Intuitively, it is much more difficult for an ML model to correctly learn all the characteristics of the minority class given its smaller sample size and skewed class distribution. A naive model can achieve reasonable accuracy by simply predicting the majority class in all cases.
An evaluation metric simply evaluates the efficiency and accuracy of a specific model in making the correct predictions. For classification problems, an evaluation metric compares the model predicted class label (or probability thereof) with the actual class label and spits out the accuracy of the prediction. There are several standard evaluation metrics that work perfectly well on balanced classification problems (e.g., Accuracy) but fail miserably when it comes to skewed datasets.
Therefore, we should utilize certain specific metrics that are more suited for imbalanced classification problems.
Before going further, let us go over the various types of evaluation metrics as classified by Cesar Ferri, et al.1:
Threshold Metrics, based on a threshold or a qualitative understanding of error: Accuracy, Sensitivity/Recall, Specificity, Precision, F-Score, Geometric Mean of Sensitivity and Specificity, and Kappa statistic.Ranking Metrics are based on how well the model ranks the classes (the concept of separatability): Area Under the Receiver Operating Characteristic curve (AUROC) and Area Under the Precision-Recall curve (PR AUC).Probability Metrics measure the deviation from the true class probabilities: LogLoss and Brier Score. Note that predicted probabilities need to be calibrated before utilizing these metrics.
Threshold Metrics, based on a threshold or a qualitative understanding of error: Accuracy, Sensitivity/Recall, Specificity, Precision, F-Score, Geometric Mean of Sensitivity and Specificity, and Kappa statistic.
Ranking Metrics are based on how well the model ranks the classes (the concept of separatability): Area Under the Receiver Operating Characteristic curve (AUROC) and Area Under the Precision-Recall curve (PR AUC).
Probability Metrics measure the deviation from the true class probabilities: LogLoss and Brier Score. Note that predicted probabilities need to be calibrated before utilizing these metrics.
Although it is the most widely-used and interpretable metric for classification problems, please do not make the rookie mistake of using the Accuracy metric to evaluate your classification model in case of imbalanced datasets.
Accuracy is calculated as:
Or, in terms of the Confusion Matrix:
Consider a highly skewed dataset with 1:100 class imbalance — for each instance of minority class (positive), there are 100 samples of the majority class (negative). A naive classifier (or, even a human being) can achieve a 99% prediction accuracy by predicting the majority class by default for each and every test sample, without any actual machine learning involved. However, this does not paint the true picture as may be correctly predicting the minority class is more contextually critical, e.g., in fraud detection or spam filtering algorithms.
Ok, so then which metric(s) should we use to evaluate our classifier? Let us go over some suitable metrics for imbalanced classification now.
Precision
Precision is the ratio of true positives to the total positives predicted by a classifier. Remember that positives, in the context of an imbalanced classification problem, refer to the minority class. It is calculated as follows in terms of the confusion matrix:
Precision values range between 0 and 1. Since it does not concern itself with false negatives and aims to minimize the number of false positives, Precision is not truly representative of the entire picture where we are also concerned with minimizing false negatives (e.g., predicting a transaction as non-fraudulent when, in fact, it is fraudulent).
Scikit-learn’s precision_score function calculates Precision for both binary and multi-class classification problems:
from sklearn.metrics import precision_score# calculate precision for binary classification problemprecision = precision_score(y_true, y_pred, average='binary')# calculate precision for multi-class classification problemprecision = precision_score(y_true, y_pred, labels=[1,2], average='micro')# labels is a list of all possible class labels
Recall
Also known as Sensitivity and True Positive Rate (TPR), Recall is the ratio of true positives to the ground-truth positives in the sample. Unlike Precision, Recall also considers the number of positive (minority) cases that were not classified as such. It is calculated as follows in terms of the confusion matrix:
Recall values range between 0 and 1. Recall is not concerned with the number of false positives and instead, aims to minimize the number of false negatives.
Scikit-learn’s recall_score function calculates Recall for both binary and multi-class classification problems in exactly the same way as precision_score.
F-Score
Precision is appropriate when we are more concerned with minimizing false positives, while Recall is appropriate when the number of false negatives is more critical.
But what if we want to maximize both Precision and Recall simultaneously to accurately predict the positive class with minimal false positives and false negatives. For example, in a fraud detection algorithm, we want to accurately predict all fraudulent cases and at the same time, not predict non-fraudulent cases as fraudulent. Maximizing both Precision and Recall together is a fine balancing act, and this is where the F-Score comes in.
F-Score provides a single score to measure both Precision and Recall. Accordingly, maximizing F-Score will mean that we would have maximized both Precision and Recall. F-Score is a simple harmonic mean of Precision and Recall and is widely used for imbalanced classification problems.
Again, F-Score ranges between 0 and 1. Just like Precision and Recall, scikit-learn’s f1_score function calculates the F-Score for both binary and multi-class problems in exactly the same way and syntax.
Area Under Receiver Operating Characteristic curve (AUROC)
The ROC curve plots a binary classifier’s False Positive Rate (FPR) on the x-axis and True Positive Rate (TPR, Recall) on the y-axis with for all possible probability threshold values between 0 and 1. TPR is calculated as follows:
The default probability threshold value for any classifier is usually 0.5, that is, classify a sample as belonging to the positive class if its predicted probability is greater than 0.5. However, this default assumption should not be used for imbalanced datasets, more on this later.
The perfect classifier should be able to maximize the TPR to 1 and minimize FPR to 0, hence the perfect classifier will be as high and up in the top-left quadrant of the ROC curve (coordinate 0,1). ROC is not particularly biased towards either the majority or minority class and is, therefore, a popular choice for both balanced and imbalanced classification problems.
A convenient way to compare two classifiers is to calculate the area under the ROC curve, AUROC, which gives a single score for a classifier across all the probability threshold values. AUROC ranges between 0 and 1, with 1 being the score of a perfectly skilled classifier.
A complete example of drawing the ROC curve and calculating AUROC follows, using scikit-learn’s roc_curve and roc_auc_score functions:
The output will be similar to like this:
ROC Threshold Moving for Imbalanced Classification
As alluded to above, using the default probability threshold of 0.5 to predict class labels in case of imbalanced classification problems will likely lead to poor model performance. Luckily, it is pretty straightforward to determine the optimal probability threshold in the case of ROC curves. Intuitively, this optimal threshold is closest to the top-left corner of the ROC curve and can be easily determined using the Youden’s J Statistic.
The J Statistic is calculated for each threshold value returned by roc_curve with the largest J value being the optimal threshold. The formula to calculate the J Statistic in its simplest form is:
Once we have called the roc_curve function, determining the optimal threshold using the J Statistic is pretty straightforward:
#import the required libraryfrom numpy import argmax# calculate inputs for the roc curvefpr, tpr, thresholds = roc_curve(test_y, yhat_positive)# get the best thresholdJ = tpr - fprix = argmax(J)best_thresh = thresholds[ix]print('Best Threshold: %f' % (best_thresh))
All samples with predicted probabilities (yhat) greater than this threshold should then be assigned to the positive class.
Area Under the Precision-Recall curve (PR AUC)
A PR curve plots Recall on the x-axis against Precision on the y-axis for all the possible probability thresholds. In contrast with the ROC curve, a perfectly skilled model bows towards the top right axis with coordinates (1,1). Since both Precision and Recall are concerned with true positives (the minority class), it makes it an effective tool for imbalanced classification problems.
A convenient way to compare two classifiers is to calculate the area under the PR curve, PR AUC, which gives a single score for a classifier across all the probability threshold values. PR AUC ranges between 0 and 1, with 1 being the score of a perfectly skilled classifier.
A complete example of drawing the PR curve and calculating PR AUC follows, using scikit-learn’s precision_recall_curve and auc functions:
The output will be similar to like this:
PR Curve Threshold Moving for Imbalanced Classification
Similar to the ROC curves, it is simple enough to determine the optimal probability threshold in the case of PR curves. Intuitively, this optimal threshold is closest to the top-right corner of the PR curve and can be easily determined using the F-Score.
Once we have called the precision_recall_curve function, the optimal threshold using the F-Score is determined as follows:
#import the required libraryfrom numpy import argmax# calculate inputs for the roc curveprecision, recall, thresholds = precision_recall_curve(test_y, yhat_positive)# Calculate F-Scores and find the index of ideal scorefscore = (2 * precision * recall) / (precision + recall)ix = argmax(fscore)best_thresh = thresholds[ix]print('Best Threshold: %f' % (best_thresh))
All samples with predicted probabilities (yhat) greater than this threshold should then be assigned to the positive class.
Log Loss Score
Log Loss Score is one of the probability-based evaluation metrics that is not concerned with the predicted class labels. Instead, it is used to evaluate the predicted calibrated probabilities. Some models predict calibrated probabilities as part of the training process (e.g., logistic regression),but many do not, and will require calibration (e.g., support vector machines, decision trees, and neural networks) before using LogLoss or Brier Scores. Probabilities can be easily calibrated using scikit-learn’s CalibratedClassifierCV class.
The best possible Log Loss Score is 0 and ranges from 0 to infinity, the higher the worse score. It is defined as the negative log-likelihood of a classifier that predicts probabilities (yhat) based on the ground true class labels (y):
Scikit-learn’s log_loss function makes it easy to calculate the Log Loss Score once we have the predicted class probabilities as follows:
from sklearn.metrics import log_loss# fit a modelmodel = LogisticRegression()model.fit(train_X, train_y)# predict probabilitiesyhat = model.predict_proba(test_X)# calculate and print Log Loss Scorelogloss = log_loss(test_y, yhat)print('Log Loss Score: %.3f' % (logloss))
Brier Score
The Brier Score, named after Glenn Brier, calculates the mean squared error between predicted probabilities and their respective positive class values. Brier Score is usually applicable to only binary classification problems and evaluates the probabilities of the positive class. The critical difference between the Brier and Log Loss scores is that the predicted probabilities of only the positive class are considered in the Brier Score.
Contrary to the Log Loss Score, a Brier Score ranges between 0 and 1, with 0 being the score of a perfectly skilled classifier. It is calculated as follows, where yhat is the predicted probability of the positive class and y is the ground-truth label of the positive class:
Scikit-learn’s brier_score_loss function makes it easy to calculate the Brier Score once we have the predicted positive class probabilities as follows:
from sklearn.metrics import brier_score_loss# fit a modelmodel = LogisticRegression()model.fit(train_X, train_y)# predict probabilitiesyhat = model.predict_proba(test_X)# retrieve the probabilities for the positive classyhat_positive = yhat[:, 1]# calculate and print Brier Scorebrier = brier_score_loss(test_y, yhat_positive)print('Brier Score: %.3f' % (brier))
The k-fold cross-validation technique is quite efficient and effective in assessing a classifier for balanced or even slightly skewed datasets. However, it fails in case of imbalanced datasets since the training data is usually split into folds assuming a uniform probability distribution. In the case of imbalanced data, this has the potential of resulting in certain folds either completely missing out on the positive (minority) class or having very few examples of it. Accordingly, this is likely to result in misleading model evaluations.
A modified k-fold cross-validation, called stratified k-fold cross-validation, is more suitable for imbalanced classification problems. Stratified k-fold or Repeated Stratified k-fold preserves the imbalanced class distribution in each fold. Either of these k-fold techniques can then be used in conjunction with cross_val_score to predict and evaluate each fold. All the evaluation metrics explained earlier can be used as a scoring parameter in cross_val_score. Refer to the full list here. A practical demonstration is included later in the section of cost-sensitive learning.
The perceived cost of misclassification errors by a classifier is usually not the same in case of an imbalanced classification problem. That is, the cost of missing a positive (minority) class is usually more critical than the incorrect classification of the negative (majority) class. In other words, false negatives are more critical than false positives.
For example, denying a loan to a good customer is not as bad as giving a loan to a bad customer that may never repay it. Or, it is better to diagnose a healthy patient with cancer and follow-up with more medical tests than it is to discharge a patient that actually has cancer.
Cost-sensitivity learning is a subfield of ML that takes into account the cost of various errors (including misclassification costs) while minimizing the model error during model training.
Cost-sensitive learning for imbalanced classification is focused on first assigning different costs to the types of misclassification errors that can be made, then using specialized methods to take those costs into account.
But how to assign these costs to misclassifications, specifically, false positives and false negatives? True positives and true negatives are usually assigned nil costs. Since false negatives are usually more critical, a higher cost is assigned to false negatives than false positives. Domain knowledge helps us here in certain cases. In an insurance claim scenario, the cost for false positives could be the potential monetary cost of customer follow-up, and the cost of a false negative might be the potential insurance claim cost.
However, assigning misclassification costs might not be straightforward in all cases. In such scenarios, an effective starting point could be to assign the inverse of the class distribution ratio to false positives and negatives. For example, if we have a dataset with a 1:100 class balance, false negatives are assigned a cost of 100 while false positives are assigned a cost of 1. This assumes that the same class distribution will also hold true in real-life when making predictions.
Now, let us go through certain ML models and see how this cost-sensitive learning can be incorporated therein.
Scikit-learns’ standard Logistic Regression model can be easily modified to consider class imbalances through the class_weight parameter. This parameter penalizes the model less for errors made on examples from the majority class and penalizes the model more for errors made on examples from the minority class.
The class_weight parameter takes in a dictionary of all possible class labels and their associated weights/cost to be considered during the model fitting. Again, these class weights can either be determined by a domain expert, through grid search, industry best practices or using the inverse of the class distribution as explained earlier.
The last option above to use the inverse class distribution as weights is specified by passing balanced as the input to the class_weight parameter instead of manually calculating the distribution.
A complete example of using a class-weighted logistic regression model and evaluating using repeated stratified k-fold follows:
Class imbalance is taken into account in decision trees by considering the importance of each class while determining the split point at each node.
Just like logistic regression, scikit-learn’s DecisionTreeClassifier class has the class_weight parameter that functions exactly like that in logistic regression.
A complete example of using a class-weighted decision tree and evaluating it using repeated stratified k-fold follows:
XGBoost, an ensemble of multiple decision trees, is a highly efficient implementation of stochastic gradient boosting. It is widely used for both classification and regression problems and has been found to be highly effective in several challenging machine learning problems.
XGBoostClassifier’s scale_pos_weight parameter is used to train a class-weighted XGBoost classifier for imbalanced data. XGBoost’s documentation provides a convenient way to estimate the ideal value for scale_pos_weight as follows:
A complete example of using class-weighted XGBoostClassifier and evaluating it using repeated stratified k-fold follows:
Generally speaking, I am not a huge fan of using various over and undersampling methods to treat class imbalance in a training dataset. My preference is to use the appropriate evaluation metrics on cost-sensitive ML algorithms before even trying any of the following sampling methods to rectify the class imbalance.
First, let’s identify the three main types of sampling methods to remedy class imbalance:
Oversampling includes duplicating or synthesizing new samples from the minority classUndersampling includes deleting or selecting only a sample of examples from the majority classSynthetic Methods combine both oversampling and undersampling
Oversampling includes duplicating or synthesizing new samples from the minority class
Undersampling includes deleting or selecting only a sample of examples from the majority class
Synthetic Methods combine both oversampling and undersampling
Reasons why I generally do not prefer sampling methods over cost-sensitive learning:
Potential overfitting in case of oversampling, however, some of the advanced techniques as explained later try to take care of thisLoss of useful data with undersamplingIncreased learning and processing time after oversampling
Potential overfitting in case of oversampling, however, some of the advanced techniques as explained later try to take care of this
Loss of useful data with undersampling
Increased learning and processing time after oversampling
If you do decide to proceed with sampling methods, then be very careful in its implementation. Sampling should only be performed on the training dataset. It is not performed on the entire dataset, holdout set, or validation dataset. The reason is that the intent is not to remove the class bias from the data but to continue to evaluate the resulting model on data that is both real and expected in production, and representative of the actual real-life problem.
Let us now have a brief look at some of the major oversampling methodologies. Note that I will not be covering the random oversampling technique (RandomOverSampler) as I consider it more prone to overfitting.
Synthetic Minority Oversampling Technique (SMOTE)
SMOTE synthesizes new data samples from the positive (minority) class without a random duplication and was introduced by Nitesh Chawla, et al. in their 2002 paper2. From the paper: “The minority class is over-sampled by taking each minority class sample and introducing synthetic examples along the line segments joining any/all of the k minority class nearest neighbors.”.
Python’s imbalanced-learn library includes an implementation of SMOTE in the aptly titled class: SMOTE (together with a host of other sampling methods described here). The SMOTE class is implemented like any other data transform object in scikit-learn: it must be defined and configured, fit on a dataset, then applied to create a new transformed version of the dataset.
Borderline-SMOTE
Borderline-SMOTE, proposed by Hui Han, et al.3 in 2005, is an extension to SMOTE which identifies and then synthesizes borderline instances of the minority class using k Nearest Neighbors algorithm. BorderlineSMOTE class of imblearn library implements Borderline-SMOTE similar to SMOTE.
Borderline-SMOTE SVM
Borderline-SMOTE SVM is identical to Borderline-SMOTE but differs in the technique used to identify the borderline cases, where an SVM algorithm is used instead of KNN to identify misclassified borderline cases. It was proposed by Hien Nguyen, et al. in their 2011 paper4. SVMSMOTE class of imblearn library implements Borderline-SMOTE SVM similar to SMOTE.
Adaptive Synthetic Sampling (ADASYN)
ADASYN was proposed by Haibo He, et al.5 in their 2008 paper. From the paper: “ADASYN is based on the idea of adaptively generating minority data samples according to their distributions: more synthetic data is generated for minority class samples that are harder to learn compared to those minority samples that are easier to learn.”. ADASYN class of imblearn library implements ADASYN similar to SMOTE.
Practical Demonstration
A complete practical demonstration of using the above four oversampling techniques with a DecisionTreeClassifier in a Pipeline using RepeatedStratifiedKFold follows (note that certain parameters of these classes can be tuned):
Let us now have a brief look at some of the major undersampling methodologies. Note that I will not be covering the random undersampling (RandomUnderSampler) technique as I consider it more likely to result in loss of useful data.
Near Miss Undersampling
Near Miss refers to a collection of three undersampling methods that select examples based on the distance between majority and minority class instances. The approaches were proposed by Jianping Zhang and Inderjeet Mani in their 2003 paper6. The three methods are:
NearMiss-1 undersamples only those majority class instances with a minimum average distance to a predetermined number of closest minority class instances
NearMiss-2 undersamples only those majority class instances with a minimum average distance to a predetermined number of furthest minority class instances
NearMiss-3 undersamples only those majority class instances with minimum distance to each minority class instance
NearMiss class of imblearn library implements all three versions of NearMiss similar to SMOTE.
Other Undersampling Methods
There are several other undersampling methods included within the imblearn library as follows that are implemented in a similar fashion:
The Condensed Nearest Neighbor Rule7 — CondensedNearestNeighbour
Two modifications of CNN8 — TomekLinks
One-Sided Selection9 — OneSidedSelection
Edited Nearest Neighbors10 — EditedNearestNeighbors
Neighborhood Cleaning Rule11 — NeighborhoodCleaningRule
It can sometimes be more effective to combine both oversampling and undersampling methods together. Just like the demonstration of SMOTE above with a Pipeline, a similar Pipeline can also be used to create a sequence of multiple sampling methods with a classifier at the end. This pipeline is then used in conjunction with RepeatedStratifiedKFold and cross_val_score to evaluate predictions.
A simple demonstration of combining SMOTE and Random Oversampling in a pipeline is:
# import required librariesfrom imblearn.pipeline import Pipelinefrom imblearn.over_sampling import SMOTEfrom imblearn.under_sampling import RandomUnderSamplerfrom sklearn.model_selection import cross_val_scorefrom sklearn.model_selection import RepeatedStratifiedKFoldfrom sklearn.tree import DecisionTreeClassifier# define modelmodel = DecisionTreeClassifier()# define pipelineoversample = SMOTE(sampling_strategy=0.1)undersample = RandomUnderSampler(sampling_strategy=0.5)steps = [('o', oversample), ('u', undersample), ('m', model)]pipeline = Pipeline(steps=steps)# define cross-validation procedurecv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)# evaluate modelscores = cross_val_score(pipeline, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
However, instead of these manual combinations, imblearn has a couple of standard synthetic implementations as follows:
SMOTE and Tomek Links — SMOTETomek
# import required librariesfrom imblearn.pipeline import Pipelinefrom imblearn.combine import SMOTETomekfrom imblearn.under_sampling import TomekLinksfrom sklearn.model_selection import cross_val_scorefrom sklearn.model_selection import RepeatedStratifiedKFoldfrom sklearn.tree import DecisionTreeClassifier# define modelmodel = DecisionTreeClassifier()# define sampling strategyresample = SMOTETomek(tomek=TomekLinks(sampling_strategy='majority'))# define pipelinepipeline = Pipeline(steps=[('r', resample), ('m', model)])# define cross-validation procedurecv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)# evaluate modelscores = cross_val_score(pipeline, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
SMOTE and Edited Nearest Neighbors — SMOTEENN
# import required librariesfrom imblearn.pipeline import Pipelinefrom imblearn.combine import SMOTEENNfrom sklearn.model_selection import cross_val_scorefrom sklearn.model_selection import RepeatedStratifiedKFoldfrom sklearn.tree import DecisionTreeClassifier# define modelmodel = DecisionTreeClassifier()# define pipelinepipeline = Pipeline(steps=[('r', SMOTEENN()), ('m', model)])# define cross-validation procedurecv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)# evaluate modelscores = cross_val_score(pipeline, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
I hope the above discussion on classification problems with imbalanced datasets would prove to be useful to you, just like it has for me.
Feel free to reach out to me to discuss anything related to machine learning or data and financial analytics.
Learn on!
Inspirations from and credit to Dr. Jason Brownlee of Machine Learning Mastery who has been instrumental in my ML journey
[1] Ferri, C., Hernández-Orallo, J., & Modroiu, R. (2009). An experimental comparison of performance measures for classification. Pattern Recognit. Lett., 30, 27–38.
[2] Chawla, N., Bowyer, K., Hall, L. & Kegelmeyer, W. (2002). SMOTE: Synthetic Minority Over-sampling Technique. J. Artif. Intell. Res. (JAIR). 16. 321–357. 10.1613/jair.953.
[3] Han, H., Wang, WY., & Mao, BH. (2005) Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning. In: Huang DS., Zhang XP., Huang GB. (eds) Advances in Intelligent Computing. ICIC 2005.
[4] Nguyen, H., Cooper, E. & Kamei, K. (2011). Borderline over-sampling for imbalanced data classification. International Journal of Knowledge Engineering and Soft Data Paradigms. 3. 4–21. 10.1504/IJKESDP.2011.039875.
[5] Haibo, H., Yang, B., Garcia, E. A. & Li, S. (2008). “ADASYN: Adaptive synthetic sampling approach for imbalanced learning,” 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, 2008, pp. 1322–1328, doi: 10.1109/IJCNN.2008.4633969.
[6] Zhang, J. & Mani, I. (2003). KNN Approach to Unbalanced Data Distributions: A Case Study Involving Information Extraction. Proceedings of the ICML’2003 Workshop on Learning from Imbalanced Datasets.
[7] Hart, P., “The condensed nearest neighbor rule,” In Information Theory, IEEE Transactions on, vol. 14(3), pp. 515–516, 1968.
[8] Tomek, I., “Two modifications of CNN” In Systems, Man, and Cybernetics, IEEE Transactions on, vol. 6, pp 769–772, 2010.
[9] Kubat, M. & Matwin, S., “Addressing the curse of imbalanced training sets: one-sided selection” In ICML, vol. 97, pp. 179–186, 1997.
[10] Wilson, D., Asymptotic “ Properties of Nearest Neighbor Rules Using Edited Data” In IEEE Transactions on Systems, Man, and Cybernetrics, vol. 2 (3), pp. 408–421, 1972.
[11] Laurikkala, J., “Improving identification of difficult small classes by balancing class distribution” Springer Berlin Heidelberg, 2001.