How I solved a class imbalance problem
Tags: Machine Learning, Class Imbalance, Data Science, Python, AI

How I solved a class imbalance problem

Using distplot() and count functions

Imbalance classifications seems to be a common problem that needs to be addressed whenever endeavouring to solve a machine learning problem. If the classes are severely imbalance, and measures are not taken to correct this, a high overall accuracy can still be achieved without generating any good insights into the problem.

I therefore endeavour to look at the target variable and analyse it whenever tackling a machine learning problem. To illustrate my point, I have selected a class imbalance competition problem from the Analytics Vidhya site, which can be found in the link below:- https://datahack.analyticsvidhya.com/contest/janatahack-hr-analytics/#MySubmissions

Extracts from the problem statement of this competition problem state:-

“A training institute which conducts training for analytics/ data science wants to expand their business to manpower recruitment (data science only) as well.

Company gets large number of signups for their trainings. Now, company wants to connect these enrollees with their clients who are looking to hire employees working in the same domain. Before that, it is important to know which of these candidates are really looking for a new employment. They have student information related to demographics, education, experience and features related to training as well.

To understand the factors that lead a person to look for a job change, the agency wants you to design a model that uses the current credentials/demographics/experience to predict the probability of an enrollee to look for a new job.”

The datasets that need to be downloaded and read for this problem have been saved to my github repository, and can be found at the link below:- https://github.com/TracyRenee61/HR-Analytics-Job-Change

The first thing I did was to import the libraries and import and read the files needed to be used in the competition question:-

I then checked for any missing values in both the train and test files, finding there were many cells that needed to be imputed:-

I then imputed the missing values by replacing all null cells in the train and test sets with mode, being the most commonly occurring value in a column.

When all of the missing values had been imputed, I converted the object columns to text by using LabelEncoder():-

I used distplot() to graphically represent the two classes in the target and this is where it was discovered there exists a class imbalance. I used value_counts() to discover that only 13.21% of the examples were 1:-

I then defined X, y, and X_test. X is the train file less the target and the enrollee_id, y is the target in the train file, and X_test is the test file less the enrollee_id.

After defining X and y, I split the train set up for training and validation using train_test_split. Because there was a class imbalance, I set stratify to y and shuffle to True.

I then put the datasets derived from train_test_split into a standard scaler to ensure the dependent variables are in the same range as the dependant.

Because the target is class imbalance, I created a variable called class_weights, where I have endeavoured to balance the classifications:-

I decided to use distplot() to have a look at y_val and found that 13.24% of the examples were 1. Therefore, any model selected would ideally pick up 13.24% of 1 in its prediction process. Let’s see what I found:-

I selected eight models and used them as the basis for this competition question. All eight of the models’ performance is illustrated in the boxplot below:-

I selected seven of the eight model to test on the validation set and the chart below shows the percentages that were achieved. This exercise proves it is a good idea to write down the results of any tests because they can be referred to at a later date for clarification. I decided that XGBClassifier was the best model to use because it gave a more accurate representation than the other models I had tested on:-

After testing the selected model on the validation set, I decided to predict on the test dataset and found that 8% of the examples were 1, being 5% less than what has appeared on the train dataset:-

Because the competition question stated probabilities had to be predicted on, I used predict_proba() to find the probability that the enrollee would be looking for a job, being a 1.

When I submitted my probability predictions to Analytics Vidhya, I achieved a score of 65.24% on the AUC_ROC score, with the highest on the leaderboard being 69.43%

The code for this blog post can be found in its entirety in my github account, found below:- https://github.com/TracyRenee61/HR-Analytics-Job-Change/blob/master/HR_Analytics_Job_Change_XGB.ipynb