Imbalanced Classes: Part 2. Recently, I wrote this post about... | by Becca R | Towards Data Science

Recently, I wrote this post about imbalanced class sizes in classification models might lead to overestimation of a classification model’s performance. The post discussed a classification project I was developing using Airbnb first user booking data from Kaggle. The objective of the project was to predict whether a first-time Airbnb user would likely book a vacation home in the U.S.A./Canada or somewhere internationally. However, bookings within the U.S.A./Canada accounted for around 75 percent of the data, making it difficult to accurately estimate international bookings.
To account for the fact that nearly 100 percent of observations were predicted to be in the dominant class (in my case, travelers to the U.S.A./Canada), I used the Adaptive Synthetic (ADASYN) to oversample the minority class in my test set. Not satisfied with the results of the out-of-box logistic regression with default parameters, I decided to do some model selection and brute-force parameter tuning using GridSearchCV in scikit-learn. My features were engineered, my classes were balanced, and my shiny new AWS instance was compute-optimized. What could go wrong?
Data pre-processing
I was eager to fit some models and see how well I could classify user location based on the features I had acquired. I had split the dataframe into my target variable (y) and a matrix of features (X). I performed train-test-split on the data set (80% training, 20% testing) with stratification to ensure that the minority class were represented in the test set. I used a standard scalar transformation on the features sets (X_train and X_test). Finally, I used the the Adaptive Synthetic (ADASYN) method to perform oversampling of the minority class in the training data (see Imbalance Classes Part I for details).
from sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import StandardScalerfrom imblearn.over_sampling import ADASYNX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,                                                 stratify=y,random_state = 88)std_scale = StandardScaler()X_train_scaled = std_scale.fit_transform(X_train)X_test_scaled = std_scale.transform(X_test)adasyn = ADASYN(random_state=88)X_adasyn, y_adasyn = adasyn.fit_resample(X_train_scaled, y_train)
GridSearchCV
I looped through five classifiers: Logistic Regression, K-Nearest Neighbors, Decision Tree, Random Forest, and Support Vector Classifier. I defined “models” to be a list of dictionaries for each classifier with the classifier object (random state set always to 88 for reproducibility, can you guess my favorite number?), and a grid of model-specific hyperparameters to be tuned.
from sklearn.linear_model import LogisticRegressionfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.tree import DecisionTreeClassifierfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.svm import SVCmodels = [{'name': 'logreg','label': 'Logistic Regression',           'classifier': LogisticRegression(random_state=88),           'grid': {"C":np.logspace(-3,3,7), "penalty":["l1","l2"]}},                    {'name': 'knn','label':'K Nearest Neighbors',           'classifier':KNeighborsClassifier(),           'grid': {"n_neighbors":np.arange(8)+1}},                    {'name': 'dsc','label': 'Descision Tree',            'classifier': DecisionTreeClassifier(random_state=88),           'grid': {"max_depth":np.arange(8)+1}},                    {'name': 'rf', 'label': 'Random Forest',           'classifier': RandomForestClassifier(random_state=88),           'grid': {'n_estimators': [200, 500],'max_features': ['auto', 'sqrt', 'log2'],                    'max_depth' : [4,5,6,7,8],'criterion' :['gini', 'entropy']}},                    {'name': 'svm_rbf', 'label': 'SVC (RBF)',           'classifier':SVC(random_state=88),           'grid': {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']}}]
I defined the model_selection function below to perform a GridSearch to optimize hypterparameters across five cross-validation sets for a given model. I decided to use the Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) score to determine model performance so as to try and maximize true positives while minimizing false positives in model predictions. The function returns a dictionary that includes the classifier, optimal parameters from the GridSearch, and the best averaged ROC_AUC score from the validation sets.
from sklearn.metrics import roc_auc_scoredef model_selection(classifier, name, grid, X_train, y_train, scoring):        gridsearch_cv=GridSearchCV(classifier,                                grid,                               cv=5,                                scoring = scoring)        gridsearch_cv.fit(X_adasyn, y_adasyn)        results_dict = {}        results_dict['classifier_name'] = name        results_dict['classifier'] = gridsearch_cv.best_estimator_    results_dict['best_params'] = gridsearch_cv.best_params_    results_dict['ROC_AUC'] = gridsearch_cv.best_score_        return(results_dict)results = []for m in models:        print(m['name'])        results.append(fit_first_model(m['classifier'],                                    m['name'],                                   m['grid'],                                   X_adasyn,                                    y_adasyn,                                    'roc_auc'))          print('completed')
Finally, I had the results of each fitted classifier to compare! This process does not take a trivial amount of time, so be prepared to have some time to kill (my kitchen was spotless by the end of this process!).
I put the results into a dataframe for better viewability:
results_df = pd.DataFrame(results).sort_values(by='ROC_AUC', ascending = False)
Well, I was quite pleased with the first row, the Random Forest Classifier, achieving a 0.85 ROC_AUC score. This got me thinking about some grand plans for an XGBoost tree classifier. I was going to get a predictive model to rival the winners of the Kaggle competition...or was I?
Eager to confirm the performance of the Random Forest Classifier with tuned hyperparameters, I scored the predictive performance of the model on my test set. I was aghast when I saw that my the ROC_AUC score for the test set was only 0.525. In my experience test scores are typically lower than cross-validation scores, and I know that Random Forest can be prone to overfitting, but this was a performance decrease of 38 percent!
For good measure, I scored the performances of the remaining four classifiers on the test set. Sure enough, the ROC_AUC test scores were significantly lower than the cross validation ROC_AUC averages. Logistic regression was an exception, performing only slightly better than a random guess on both the validation and test sets.
Order matters
What was going on here? Well, remember that I oversampled with ADASYN before splitting my training data for cross-validation. So, my five-fold validation sets are NOT representative of a distribution in the real world. Rather, they contain “synthetic” data points representative of hard-to-classify observations in the minority class. Therefore, when I scored the model performance on the test set (with the target class proportions indicative of the real-world), the score dropped significantly.
Luckily, imbalanced-learn has a Pipeline class that will apply the ADASYN resampling only during the classifier fitting, thus allowing me to avoid some clunky for-loops and manual GridSearchCV.
Below is the code to build the pipeline for GridSearchCV hyperparameter tuning on the Random Forest Classifier with oversampling during cross-validation fitting. (Note the class__ prefix in the grid dictionary!)
from imblearn.pipeline import make_pipeline, Pipelinerf = RandomForestClassifier(random_state=88)adasyn = ADASYN(random_state=88)grid = {'class__n_estimators': [200, 500],        'class__max_features': ['auto', 'sqrt', 'log2'],        'class__max_depth' : [4,5,6,7,8],        'class__criterion' :['gini', 'entropy']}pipeline = Pipeline([('sampling', adasyn), ('class', rf)])grid_cv = GridSearchCV(pipeline, grid, scoring = 'roc_auc', cv = 5)   grid_cv.fit(X_train_scaled, y_train)grid_cv.best_score_
Lo and behold, the cross-validation ROC_AUC was only 0.578, much more indicative of the model’s actual performance when applied to the test set. Once again, I looped through all models and saved optimal parameters, average validation scores, and test scores for completeness.
Key takeaways
The ROC-AUC curves for classification algorithms with synthetic data in the validation sets (left) demonstrate a clear overestimation of fit when compared to the ROC-AUC curves scored on a validation set with representative imbalanced classes (right). (Note: These algorithms were scored on a single validation set, rather than an average of three-folds, hence the difference in scores from the tables above).
While I am no closer to being able to confidently predict an Airbnb user’s vacation destination, this project illuminated the importance of paying attention to what’s happening “under the hood” of advanced machine learning algorithms. In particular, I built on my previous post about how imbalanced class sizes can lead to over-estimation of a classifier’s performance to discuss why order matters when fitting a classifier on re-sampled training data with cross-validation and hyperparameter tuning.
I hope that you found this series useful. Please feel free to provide any comments or additional insights in the comments below!