Predicting Customer Churn
By: Chris Rectenwald
Tags: Data Science, Machine Learning, Gradient Boosting, Exploratory Data Analysis, Spark

Photo by NeONBRAND on Unsplash

An important metric for the subscription based business model is a customer’s churn rate. This is when a customer decides to no longer pay for the business’s service. Understanding the reasons for these customers leaving your service can provide insights into a business’s operations as well as can dictate whether or not a business fails. In Udacity’s Data Scientist nanodegree program, I was given the opportunity to use data related to a fictitious streaming company called Sparkify, and attempt to build a predictive churn model for their company.

The Business Problem

The goal of this project was to use Sparkify’s site traffic to identify user’s who are more likely to churn than not before they actually do. This is to allow Sparkify to intervene with the customer to help navigate them through any issues that may cause them to stop paying for the service.

First I will explore the data to find insights related to a user’s subscription age, operating systems and devices, gender, and activeness on the application to name a few features that I think will be critical to predicting customer churn. From there, I aim to create a pipeline of these features to feed into a supervised learning boost tree classification algorithm. I am attracted to this algorithm due to its ability to recalculate the weighted inputs, which are useful when dealing with imbalanced data.

On second thought, I feel like if this were to be the case, the infrastructure would have to leverage some sort of data streaming tool rather than query for a user’s history each moment that one could show signs of churning from the application. With that the churn prediction will need to arrive quickly to trigger a response that can give a few days for appropriate interventions.

Exploratory Data Analysis

In this case, I was given a fairly large customer data set of 12GB. Here, the standard tools for analysis and machine learning will not fit a computer’s memory, so I am forced to user Big Data tools like Apache Spark. The data set has 18 fields which includes subscription level, userID, sessionID, gender, timestamps, author of the song, and many more.

To start, I loaded the data frame then checked out the schema and shape of the data frame. From the looks of the image below, there are around 530k entries with 19 columns.

Preparation: Some records did have either an empty userID or sessionID, so I dropped them

Now, I will look at some categorical variables in the data set: authorization, gender, method, level of subscription, method used in interaction, which page user was on, status of interaction. As one can see there are a significant amount of paid and free users that are important to Sparkify’s business as they provide revenue through paid-subscriptions and ad revenue, respectively.

First, let’s take a look at how many of users churned vs not churned.

Then, I think it’d be cool to check out churn statistics by gender or by subscription:

Based off some basic analysis, it seems that there are slightly more free users than paid users, but the cancelled paid users are about the same as the cancelled free users. It also seems that males tend to churn more than females.

Another feature that I would like to investigate is that does one’s operating system affect one’s likeliness to churn?