Handling imbalanced data:
By: Levi Raichik
Tags: Data Science, Imbalanced Data, Python, Machine Learning

Handling imbalanced data:

Predicting customers canceling phone service

In this project I am trying to predict customers that are cancelling phone service in order to do something to retain those customers.

The data-set I am using for this came from openml.org, and you can click here for the direct link to this data-set.

Imbalanced Dataset

This dataset was imbalanced which brought some challenges to it. First lets take a look at the imbalance:

Amount of customers staying or cancelling

To deal with this, I used the SMOTE method of up-sampling data. I tried using ADYSN as well, however it did not preform as well. This is probably due to ADYSN also making random noise, which for this problem, just lessened how well the models were able to predict the outcome.

Due to how the data-set is, I decided to optimize for recall, while still taking into account accuracy to a certain extent. This was due to wanting to correctly predict as many customers leaving as possible, while still holding the false positives as low as possible.

I used cost-benefit analysis to grade the models. The calculation for that was as follows: From the information I had, the average monthly bill was $60. I then calculated that over the length of a 24 month contract, which came out to be $1,440. I then decided on a promotion amount of $500, just for calculation purposes.

Any True Negatives, or customers that are correctly predicted as staying, do not add or subtract anything.

Any False Negatives, or customers that would be canceling but are predicted as staying are a loss of $1,440 each.

Any True Positives, or customer that would be canceling and are predicted as canceling, would be offered a promotion so the gain would be $1,440 - $500 = $940.

Any False Positives, or customers that are predicted as leaving but are not, would be offered the promotion and therefore would be a loss of $500.

Correlation of features

Another thing I noticed at the beginning was about the correlation of the features:

Not a lot of correlation to the target, which was the ‘class’ feature, a few at about .2, also minutes and charge are correlated almost %100. That made sense, as at the time this data was made minutes were the most important part of a bill.

Modeling

When I was contemplating the amount of features and the correlation issue, on one hand having a lot of correlation between charges and minutes but on the other hand not many dominant features. I therefore thought that random forest would work best with this type of data due to how it randomizes the features and data for each tree, then makes predictions off the entire forest. I tested a few different methods with their basic default hyper-parameters and random forest did preform the best. The results I got for my test data, after optimizing it a few times, was this:

The cost — benefit of this was $71,320 or for the actual gain to retain the customers that would be leaving would be $589 per customer that would actually have previously left, if all 121 customers that we predicted to be leaving would then stay.

After doing 5 fold cross validation the average cost-benefit for the test set was $51,152.

Cross validation

Just a note on cross validation when dealing with imbalanced data. As you should only smote training data and then predict on the real test data, that limits how you can do cross validation. The regular cross validation methods would not work with this.

In order to combat that, I used Stratified K-Folds, which takes the same amount from each category so you end up with similar splits for each fold.

Then I got each split and applied SMOTE to the training data, trained the model on that. Saved the scores for each fold then averaged them.

Here is the sample code:

skf = StratifiedKFold(n_splits=5) # Set up the splits X_trains2 = {} # Make dictionaries for each train and test split

X_tests2 = {}

y_trains2 = {}

y_tests2 = {}

for i, (train_index, test_index) in enumerate(skf.split(X, y)): # Get the folds and make the index number the key in each dict

X_trains2[i] = X.loc[train_index]

X_tests2[i] = X.loc[test_index]

y_trains2[i] = y[train_index]

y_tests2[i] = y[test_index] scores2 = [] # Make a list to put all scores into

train = 0 # Setup to save all train scores

test = 0 # Setup to save all test scores

cb = 0 # Cost-Benefit numbers for i in range(5):

smoted_x, smoted_y = sm.fit_sample(X_trains3[i], y_trains3[i]) # SMOTE the training splits

rf5.fit(smoted_x, smoted_y) # Fit the model

trainpred = rf5.predict(X_trains3[i]) # Predict train (not Smoted)

testpred = rf5.predict(X_tests3[i]) # Predict test

train += recall_score(y_trains3[i], trainpred) # Total all train recall scores for each loop

test += recall_score(y_tests3[i], testpred) # Total all train recall scores for each loop



cb += cost_benefit(y_tests3[i], testpred) # Total the Cost-benefit scores scores2.append((i, recall_score(y_trains3[i], trainpred), recall_score(y_tests3[i], testpred))) # Append a tuple of index, train recall total and test recall total) print(f’

{recall_score(y_tests3[i], testpred)}’)

cm(y_tests2[i], testpred) # Print a confusion matrix

print(classification_report(y_tests3[i], testpred))

print(train/5, test/5, cb/5) # Print the total scores / # of Folds

Feature Importance

After having a good model to work with I was then able to look at the feature importance to see what I could figure out from it with regards to bettering the interpret-ability of the model to then be able to use that for future business decisions.

I did some feature engineering to combine all the minutes and all the charges into 2 features. As those 2 features are highly correlated, I dropped the minutes column. I also dropped all call number columns, as well as the state and account_length columns as they were close to 0 on feature importance.

I also dropped number_vmail_messages as it was highly correlated with having a voice mail plan.

I ended up with 4 features, here they are with the corresponding importance:

Top Features

Running the model with only these features, with 5 fold cross validation brought the cost-benefit to $56,964.

This showed me that the model got better by combining and dropping features based on the feature importance.

Analysis of Features based on importance

After looking into features relationships, the most significant thing that jumped out at me was that anyone with a bill over $72 was much more likely to leave. That can be seen in this visual:

The numbers for over $72 bills were as follows:

Leaving: 315

Staying: 247

Considering that in the general data there is a 85% staying to 15% leaving rate, to have more people leaving at the higher rate then staying is very important.

This also was consistent in checking how customers with bills over $72 crossed over with the other features.

If we would offer a $500 promo to everyone with a average bill over $72 then per customer staying we would be gaining $547, which is similar to what we would get over the entire data-set.

I would suggest looking into this more to see how best to handle customers with higher bills to retain more of them.

Recommendations

My final recommendations would be to offer some sort of promo to customers.

To figure out how much should be offered and what sort of promos would be most successful in retaining customers.

To research more how to retain customers with higher bills.

Please leave feedback if you have thoughts on this and if you would like to see the code used for it, that can be seen here.