Understanding Customer Churning with Big Data Analytics | by Bowen Chen | Towards Data Science

Take a guess. What is the world’s most valuable asset right now?
It is not gold, not crude oil... it is data. You must have heard about the popular buzz word “big data”, and wondering what exactly that term means. Think about your favorite music streaming services — Spotify, Pandora... etc. Every second across the world, many different users login to the service, have their fair share of interactions with the service. Since every move corresponds to a collectible data point, you would imagine the challenges to store such large data.
Fortunately, these large datasets could bring real value to the business if they are being consumed in the right way. In this blog post, we will discuss a common use case for these big datasets — predicting customers churning.
Understanding Churn
Churning refers to the behavior of discreetly losing customers from a large population of customers. Some of the churned customers might turn to a competitor, while some of them might simply leave the services forever.
It might not be noticeable at the beginning, but the effect of such losses accumulates over time. As decision-makers at these music streaming service providers, we would like to understand the contributing factors of such churning behaviors — that will serve as the primary objective of our project today.
We will start by examining the dataset available.
The full dataset is a user log that is stored in a 12 GB .json file — a size that could only be handled by some big data tools, such as Spark. To fully understand the fields available, we will start by obtaining a small subset of the data (~128 MB) for exploratory data analysis purposes on a single machine. We will load in our dataset through the following commands,
# create a Spark sessionspark = (SparkSession.builder                     .master("local")                     .appName("Creating Features")                     .getOrCreate())# Read in .json file as eventsevents = spark.read.json('mini_sparkify_event_data.json')events.persist()
Spark has a great one-line shortcut of showing all the fields and their respective data types
events.printSchema()root |-- artist: string (nullable = true) |-- auth: string (nullable = true) |-- firstName: string (nullable = true) |-- gender: string (nullable = true) |-- itemInSession: long (nullable = true) |-- lastName: string (nullable = true) |-- length: double (nullable = true) |-- level: string (nullable = true) |-- location: string (nullable = true) |-- method: string (nullable = true) |-- page: string (nullable = true) |-- registration: long (nullable = true) |-- sessionId: long (nullable = true) |-- song: string (nullable = true) |-- status: long (nullable = true) |-- ts: long (nullable = true) |-- userAgent: string (nullable = true) |-- userId: string (nullable = true)
After looking at the head of the dataset, we will recognize that all of the user interactions are recorded in the page column.
While most of the users keep playing the next song, there are some users going into the “Cancel” page — and they all confirmed their cancellations through the “Cancellation Confirmation” page (see the same number of “Cancel”s and “Cancellation Confirmation”s. We will define the churning activity specifically as the number of Cancellation Confirmation. The users with page column have “Cancellation Confirmation” will be the churned users that we are specifically interested in.
Feature Engineering
We have successfully identified the churned users, now it is time to use our business mindset to think of the contributing factors to users churning. What would you do if you really hate your music streaming services? I have come up with the below 7 features,
Representative User Interactions
Representative User Interactions
We would reasonably expect some of the other user interactions will eventually lead to our user churning. We could use a box plot to perform the first level filtering. Box plots will effectively help us visualize the minimum, 25th percentile, mean, 75th percentile and maximum for particular data distribution. By plotting box plots for both churned and non-churned users, we could clearly interpret the difference between the two types of users for a particular interaction.
The following interactions showed significant distribution differences between the two groups,
Add Friends — Churned users are less likely to add friends
Add to Playlist — Churned users are less likely to add to playlists
Upgrade — Churned users have a wide range of upgrade activities
NextSong — Churned users are less likely to play next song
ThumbsUp — Churned users are less likely to press thumbs up
Roll Advert — Churned users have a wider spread on roll advert
Settings — Churned users less likely to visit the settings page
Log out — Churned users less likely to log out (due to fewer logins)
Help — Non-churned users more likely to ask for help
Home — Churned users less likely to visit the home page
Notice that all of our user interactions are in the same column, we will need to pivot and aggregate the total number of a certain interaction for each customer. Based on the above filtering, we will drop some of the less significant interactions.
events = events.drop('firstName', 'lastName', 'auth',                      'gender', 'song','artist',                      'status', 'method', 'location',                       'registration', 'itemInSession')events_pivot = (events.groupby(["userId"])                      .pivot("page")                      .count()                      .fillna(0))events_pivot = events_pivot.drop('About', 'Cancel', 'Login',                                   'Submit Registration','Register',                                 'Save Settings')
2. Average Music Play Time
For myself, I would probably be using it way shorter than normal users. Therefore, the average length of time the user spent playing music will be a really important factor. A simple visualization showed confirming results on below.
We will add this feature to our events_pivot table,
# filter events log to contain only next songevents_songs = events.filter(events.page == 'NextSong')# Total songs length playedtotal_length = (events_songs.groupby(events_songs.userId)                           .agg(sum('length')))# join events pivotevents_pivot = (events_pivot.join(total_length, on = 'userId',                                   how = 'left')                     .withColumnRenamed("Cancellation Confirmation",                                        "Churn")                     .withColumnRenamed("sum(length)",                                         "total_length"))
3. Number of active days
We would also expect a difference in the number of active days between the churned/non-churned groups. Since the datetime column only contains units in seconds, we will need to use a window function to aggregate the total active time for each customer and convert the value to days. We will add this feature to events_pivot.
convert = 1000*60*60*24 # conversion factor to days# Find minimum/maximum time stamp of each usermin_timestmp = events.select(["userId", "ts"])                     .groupby("userId")                     .min("ts")max_timestmp = events.select(["userId", "ts"])                     .groupby("userId")                     .max("ts")# Find days active of each userdaysActive = min_timestmp.join(max_timestmp, on="userId")daysActive = (daysActive.withColumn("days_active",                       (col("max(ts)")-col("min(ts)")) / convert))daysActive = daysActive.select(["userId", "days_active"])# join events pivotevents_pivot = events_pivot.join(daysActive,                                  on = 'userId',                                 how = 'left')
4. Number of days as paid users
Similarly, we could also calculate the number of days as paid users by using a window function, we only need to add a filter for the customer to be a paid customer user.
# Find minimum/maximum time stamp of each user as paid userpaid_min_ts = events.filter(events.level == 'paid')                    .groupby("userId").min("ts")paid_max_ts = events.filter(events.level == 'paid')                    .groupby("userId").max("ts")# Find days as paid user of each userdaysPaid = paid_min_ts.join(paid_max_ts, on="userId")daysPaid = (daysPaid.withColumn("days_paid",                     (col("max(ts)")-col("min(ts)")) / convert))daysPaid = daysPaid.select(["userId", "days_paid"])# join events pivotevents_pivot = events_pivot.join(daysPaid,                                 on = 'userId',                                  how='left')
5. Number of days as a free user
Now using the free user filter, we can find the number of days as a free user for each customer,
# Find minimum/maximum time stamp of each user as paid userfree_min_ts = events.filter(events.level == 'free')                    .groupby("userId").min("ts")free_max_ts = events.filter(events.level == 'free')                    .groupby("userId").max("ts")# Find days as paid user of each userdaysFree = free_min_ts.join(free_max_ts, on="userId")daysFree = (daysFree.withColumn("days_free",                         (col("max(ts)")-col("min(ts)")) / convert))daysFree = daysFree.select(["userId", "days_free"])# join events pivotevents_pivot = events_pivot.join(daysFree,                                 on = 'userId',                                 how='left')
6. Number of sessions
The number of music playing sessions could also be a contributing factor. Since the sessionId is available in this dataset, we could directly count the number of unique ids for each user with the groupby clause.
# count the number of sessionsnumSessions = (events.select(["userId", "sessionId"])                      .distinct()                      .groupby("userId")                       .count()                      .withColumnRenamed("count", "num_sessions"))# join events pivotevents_pivot = events_pivot.join(numSessions,                                 on = 'userId',                                 how = 'left')
7. User access agent
The streaming service might have disparate performances on different user agents. We will try to add this factor to the model. Since there are 56 different user agents, we will use Spark’s one-hot encoder to turn these different user agents into a vector.
# find user access agents, and perform one-hot encoding on the user userAgents = events.select(['userId', 'userAgent']).distinct()userAgents = userAgents.fillna('Unknown')# build string indexerstringIndexer = StringIndexer(inputCol="userAgent",                                    outputCol="userAgentIndex")model = stringIndexer.fit(userAgents)userAgents = model.transform(userAgents)# one hot encode userAgent columnencoder = OneHotEncoder(inputCol="userAgentIndex",                         outputCol="userAgentVec")userAgents = encoder.transform(userAgents)                    .select(['userId', 'userAgentVec'])# join events pivotevents_pivot = events_pivot.join(userAgents,                                  on = 'userId',                                 how ='left')
Model Building
After we have engineered the appropriate features, we will build three models — logistic regression, random forest and gradient boosting trees. To avoid writing redundant code, we will build stage objects and construct pipelines with a different classifier at the end of the pipeline.
# Split data into train and test setevents_pivot = events_pivot.withColumnRenamed('Churn', 'label')training, test = events_pivot.randomSplit([0.8, 0.2])# Create vector from feature datafeature_names = events_pivot.drop('label', 'userId').schema.namesvec_asembler = VectorAssembler(inputCols = feature_names,                               outputCol = "Features")# Scale each columnscalar = MinMaxScaler(inputCol="Features",                       outputCol="ScaledFeatures")# Build classifiersrf = RandomForestClassifier(featuresCol="ScaledFeatures",                             labelCol="label",                            numTrees = 50,                             featureSubsetStrategy='sqrt')lr = LogisticRegression(featuresCol="ScaledFeatures",                          labelCol="label",                         maxIter=10,                        regParam=0.01)gbt = GBTClassifier(featuresCol="ScaledFeatures",                     labelCol="label")# Consturct 3 pipelinespipeline_rf = Pipeline(stages=[vec_asembler, scalar, rf])pipeline_lr = Pipeline(stages=[vec_asembler, scalar, lr])pipeline_gbt = Pipeline(stages=[vec_asembler, scalar, gbt])# Fit the modelsrf_model = pipeline_rf.fit(training)lr_model = pipeline_lr.fit(training)gbt_model = pipeline_gbt.fit(training)
now the three objects rf_model, lr_model, gbt_model, represents the 3 different fitted models.
Model Evaluation
We will test the fitted models' performances and select the one that has the best performance as the final model. We will start by building a function especially for this purpose,
def modelEvaluations(model, metric, data):    """ Evaluate a machine learning model's performance         Input:             model - pipeline object            metric - the metric of the evaluations            data - data being evaluated        Output:            [score, confusion matrix]    """    # generate predictions    evaluator = MulticlassClassificationEvaluator(                metricName = metric)    predictions = model.transform(data)        # calcualte score    score = evaluator.evaluate(predictions)    confusion_matrix = (predictions.groupby("label")                                   .pivot("prediction")                                   .count()                                   .toPandas())    return [score, confusion_matrix]
We will call the above function to evaluate the above models
f1_rf, conf_mtx_rf = modelEvaluations(rf_model, 'f1', test)f1_lr, conf_mtx_lr = modelEvaluations(lr_model, 'f1', test)f1_gbt, conf_mtx_gbt = modelEvaluations(gbt_model, 'f1', test)
The gradient boosting model showed the best performance (F1 Score) in the test set. F1 score is defined as the harmonic mean of precision and recall, calculated as the following
Precision is computing the proportion of positive class identifications being the correct ones, in mathematical expressions it is
precision = tp/(tp + fp)
Recall is computing the proportion of actual positive class samples being identified correctly, in mathematical expressions it is,
recall = tp/(tp + fn)
You might be wondering the reason we opt for a much more complicated metric over the most intuitive accuracy — it is due to imbalance class distributions existed in the dataset. Since only a small portion of users will eventually churn, we would like our model to correctly identify them instead of pursuing a high overall performance. Imagine if only 6% of the customer will churn within the true population distributions, predicting everyone as non-churn will still give us 94% accuracy. On the other hand, F1 score will penalize poor performances on a single class, which will effectively mitigate such issues. The imbalance class characteristic will be present in every churn prediction problems — F1 will always be the metric to use in the future.
Feature Importance
We will utilize the feature importance function and visualize the relative importance rank of each feature we built. Due to the last feature userAgentVec is, in fact, a one-hot encoded vector, we will treat the userAgentVec feature as one. The below code will sum up all the feature importance values for all the sub-features obtained from the one-hot encoded vector.
feature_importances = np.array(gbt_model.stages[-1]                               .featureImportances)userAgentVec = feature_importances[len(feature_names) :].sum()feature_importances = feature_importances[:len(feature_names)]                      + [userAgentVec]
Now we plot the feature importance for gradient boosting trees.
Most of the features we constructed are quite important contributing factors for user churning, with days_active as the most important factor.
Full Dataset Run
We have constructed the appropriate framework — we are ready to follow the same steps as above to let the model run on the full 12 GB dataset using AWS’s EMR service. We will initialize the session by the following way
# Create spark sessionspark = (SparkSession         .builder         .appName("Sparkify")         .getOrCreate())# Read in full sparkify datasetevent_data = "s3n://dsnd-sparkify/sparkify_event_data.json"events = spark.read.json(event_data)
We will not repeat the steps again — I have attached the full script on the nbviewer site.
Eventually, the gradient boosting model produced an F1 score of 0.8896, which is a great performance.
+-----+----+---+|label| 0.0|1.0|+-----+----+---+|    0|1612| 70||    1| 163|344|+-----+----+---+
Business Strategy
What a journey we have been through — but we have not finished our mission yet. In the data science world, there is a business intention behind every model. With the feature importance we produced, we could definitely come up with some business strategies to counter customer churns. We will briefly discuss two possible strategies that will truly drive some values for our providers.
We know that the number of active days is the most significant factor, we could advise the upper management to construct a reward system to encourage low activity users to stay online for extended periods of time.
In addition, since the agents that users used to access the service is also quite significant, we could also find out the poor-performing agent and get our engineering team to work specifically to solve the issue.
I hope you have enjoyed reading this post as much as I enjoy writing it. Together, we used Spark, the big data analytics framework, to build an end-to-end machine learning workflow to identify potential customer churning for music streaming services from the user logs. We performed the following steps,
Exploratory Data AnalysisFeature EngineeringModel BuildingModel EvaluationsModel Scale UpBusiness Strategies
Exploratory Data Analysis
Feature Engineering
Model Building
Model Evaluations
Model Scale Up
Business Strategies
Due to the sheer size of the data we are working with, performing hyperparameters search has been particularly difficult. One could imagine leaving the notebook open for extended periods of time would not be the most efficient way to use our resources. Some of the nice tricks in using ssh to keep the program running could definitely help tackle this challenge.
If there is one thing we should take away from this piece — It comes back to the point we have mentioned at the beginning of the story. As brilliant engineers, we often get drawn to building the best high-performance models through various techniques, but our model needs to solve business problems for it to be valuable.