Predict Churn Before It Happens with PySpark
By: Joel B.
Tags: Big Data, Pyspark, Spark, Churn Prediction, Machine Learning

Predict Churn Before It Happens with PySpark

A different sort of churn. Photo by Cassie Matias on Unsplash

Managing churn, the process of losing and gaining customers over time, is critical for subscription-based services like Spotify or Netflix. At the same time, these services are capable of collecting massive amounts of data pertaining to user interactions. This provides an opportunity to apply machine learning as a means to identify vulnerable users based on their behaviour, and take action before they choose to leave or downgrade their plans.

Critically, the sheer amount of data involved requires large-scale “big data” processing techniques. Apache Spark has been designed exactly for just such tasks. It is a modern analytics engine designed to leverage distributed computing and storage systems to handle even petabyte sized datasets. It scales well — while Amazon EMR is a relatively straightforward way to deploy and manage Spark clusters, it can also run on a local machine.

Overview

In this post we look at a log of user interactions from a fictional “Sparkify” music service, which is analgous to Spotify or Deezer in that it offers paid and free-tiered plans, and users can upgrade, downgrade or cancel at any time.

PySpark, the Spark python interface, is used for initial exploration of the event data, and feature and label extraction, using the pyspark.sql library. The transformed dataset is then used to train a machine learning classifier to predict churn events. A small 128 mb subset of a larger 12 Gb datset is used, which makes it possible to learn to use Spark on a local machine rather than a cluster. This also allows us to compare the pyspark.ml machine learning library with Scikit-Learn, in order to contrast the two approaches.

For the classifier we are foremost interested in Recall — identifying all potential churn events (True Positives), while minimising the number of False Negatives. At the same time we should consider the Precision—we do not want to mis-identify too many stable users as vulnerable (False Positives) as there is likely a cost involved when taking action to prevent churn. To this end, precision and recall was balanced with the weighted F1-score metric (Fᵦ-score), with an emphasis on recall (β=2).

The code below assumes a Python 3.8+ environment and a recent version of Pyspark. The project files are available on a github repository.

The Data (and PySpark)

The first step is to extract the raw event data. In terms of code, the main difference with accessing data on a distributed filesystem is the URI. For example, hdfs:// for a Hadoop Distributed File System.

import os

from pyspark.sql import SparkSession # create the spark session

spark = SparkSession.builder.appName("Sparkify").getOrCreate() # read in the json dataset as a Spark DataFrame

# note the URI requires a full path

fin = f"file://{os.getcwd()}/mini_sparkify_event_data.json"

df = spark.read.json(fin) # Create temp view for SQL queries

df.createOrReplaceTempView("table")

df.printSchema() gives the column names and types

root

|-- artist: string (nullable = true)

|-- auth: string (nullable = true)

|-- firstName: string (nullable = true)

|-- gender: string (nullable = true)

|-- itemInSession: long (nullable = true)

|-- lastName: string (nullable = true)

|-- length: double (nullable = true)

|-- level: string (nullable = true)

|-- location: string (nullable = true)

|-- method: string (nullable = true)

|-- page: string (nullable = true)

|-- registration: long (nullable = true)

|-- sessionId: long (nullable = true)

|-- song: string (nullable = true)

|-- status: long (nullable = true)

|-- ts: long (nullable = true)

|-- userAgent: string (nullable = true)

|-- userId: string (nullable = true)

Basic statistics / SQL Interface

We can use a SQL query to get some basic statistics about the dataset:

spark.sql("""

SELECT

COUNT(DISTINCT userId) AS userId,

COUNT(DISTINCT sessionId) AS sessionId,

COUNT(DISTINCT CONCAT(artist, ' - ', song)) AS artist_song,

MIN(DATE(from_unixtime(ts/1000))) AS min_date,

MAX(DATE(from_unixtime(ts/1000))) AS max_date

FROM table

""").show() +------+---------+-----------+----------+----------+

|userId|sessionId|artist_song| min_date| max_date|

+------+---------+-----------+----------+----------+

| 226| 2354| 65416|2018-10-01|2018-12-03|

+------+---------+-----------+----------+----------+

So we have 226 users in this dataset, with about 10x as many sessions and 65416 unique songs played, over a period of around two months.

Popular Artists / Python Interface

Alternatively, we can use functions for a Pandas-like approach, in order to identify the most popular artists in the dataset:

artists = df.groupBy("Artist").count()

artists = artists.dropna().sort("count", ascending=False)

artists = artists. toPandas()

With .toPandas() pandas+matplotlib can be used to visualise results, assuming the returned dataframe is not too large.

That’s quite a mix

Operating System / PySpark SQL Functions

As PySpark is not pure python but rather an interface to Spark, many builtin functions, like sum() can not be used. Instead, pyspark.sql provides a separate set of functions. Here the regexp_extract function is used to extract information about the distribution of operating systems used from the UserAgent string stored for each event.

import pyspark.sql.functions as F # group by userId and aggregate to store the userAgent string

# creates a new column called "first(userAgent)"

os = df.groupBy("userId").agg({"userAgent": "first"}) # create a new column called "os" using regexp_extract

os = os.withColumn("os", F.regexp_extract(

F.col("first(userAgent)"), r"\((.+?)[N;\)]", 1)

) os.groupby("os").count().dropna().sort("count").show() +----------+-----+

| os|count|

+----------+-----+

| iPad| 3|

|compatible| 5|

| X11| 12|

| iPhone| 13|

| Macintosh| 86|

| Windows | 106|

+----------+-----+

Null Values / Lazy Loading

It’s not straightforward to identify all null entries, due to the way Spark handles data. Spark also implements lazy loading, which means that behind the scenes it will wait until a command like .collect() is called in order to optimise its approach before transforming the data.

# use list comprehension to move through all columns

# .alias() sets the column name

# .collect() stores the results rather than prints nans = df.select(

[F.count(F.when(F.isnan(c), c)).alias(c) for c in df.columns]

).collect() nulls = df.select(

[F.count(F.when(F.isnull(c), c)).alias(c) for c in df.columns]

).collect() empties = df.select(

[F.count(F.when(F.col(c)=="", c)).alias(c) for c in df.columns]

).collect()

These results revealed zero nans, many nulls (equivalent to None in pure python) and some empty userId entries, which are related to users that have not yet logged in.

From this point the the post will just focus on results, but the underlying code can be found in the repository

Identifying features related to churn

Here’s a sample of entries sorted by time for a user through a churn event.

+--------------------+--------------------+----------------+-----+

| artist| song| page|level|

+--------------------+--------------------+----------------+-----+

| Onar| Xehasmeni Melodia| NextSong| paid|

| Kate Voegele|You Can't Break A...| NextSong| paid|

| null| null| Downgrade| paid|

| All Out War|Truth In The Age ...| NextSong| paid|

|La Mancha De Rolando| Donde Vamos| NextSong| paid|

| Styx| Lady '95| NextSong| paid|

|The Berg Sans Nipple|Horseshoes and Ha...| NextSong| paid|

| Natiruts|Jamaica Roots II(...| NextSong| paid|

| null| null| Downgrade| paid|

| null| null|Submit Downgrade| paid|

| null| null| Home| free|

| Crystal Castles|Crimewave (Crysta...| NextSong| free|

|Red Hot Chili Pep...| Hey (Album Version)| NextSong| free|

| Claire Lynch|I Believe In Forever| NextSong| free|

| null| null| Add to Playlist| free|

| Rihanna|Don't Stop The Music| NextSong| free|

| The Clash| City Of The Dead| NextSong| free|

| The Black Keys| Too Afraid To Love| NextSong| free|

| null| null| Home| free|

| Modeselektor| Hasir| NextSong| free|

|Cheech And Chong ...|Bloat On (featuri...| NextSong| free|

| Die Mooskirchner| Frisch und g'sund| NextSong| free|

| Daler Mehndi| Boom Boom| NextSong| free|

| null| null| Thumbs Up| free|

+--------------------+--------------------+----------------+-----+

The page and level columns are important variables to consider. The user accesses the downgrade page multiple times before committing with submit downgrade. Note that null values arise for column entries that are not associated with accessed page.

Page Statistics

By looking at the unique page values we can get an idea of how users interact with the service.

+-------------------------+------+-------+

|page |count |percent|

+-------------------------+------+-------+

|NextSong |228108|79.619 |

|Home |14457 |5.046 |

|Thumbs Up |12551 |4.381 |

|Add to Playlist |6526 |2.278 |

|Add Friend |4277 |1.493 |

|Roll Advert |3933 |1.373 |

|Login |3241 |1.131 |

|Logout |3226 |1.126 |

|Thumbs Down |2546 |0.889 |

|Downgrade |2055 |0.717 |

|Help |1726 |0.602 |

|Settings |1514 |0.528 |

|About |924 |0.323 |

|Upgrade |499 |0.174 |

|Save Settings |310 |0.108 |

|Error |258 |0.09 |

|Submit Upgrade |159 |0.055 |

|Submit Downgrade |63 |0.022 |

|Cancel |52 |0.018 |

|Cancellation Confirmation|52 |0.018 |

|Register |18 |0.006 |

|Submit Registration |5 |0.002 |

+-------------------------+------+-------+

I’ve highlighted the pages that we will use as target labels (Y) for our classifier training, while most of the other pages will form the bulk of the features (X).

Commitment rates for different churn events

Users accessed the Downgrade page over 2055 times, but only actually committed to downgrading 63 times. If we also include cancel events in this process then this indicates a commitment rate of only 6%.

This gives an idea of how user interactions (i.e. visiting the downgrade page) can be used to potentially predict churn.

User phases

Users retain their same userId as they go through downgrade or upgrade events. As such, it was important to separate the phases of each userId over time, each bounded by a churn event (or the end of the log file) that identifies the downgrade/upgrade/cancel/steady churn state.

+--------------+------+-----------+---------+--------+

| userId_phase | paid | downgrade | upgrade | cancel |

+--------------+------+-----------+---------+--------+

| 447 | 208 | 63 | 159 | 52 |

+--------------+------+-----------+---------+--------+

447 unique data points were identified, split between 208 paid and 239 free groups. Class imbalance could be an issue for classifier training as only 115 cancel/downgrade examples are present.

We can now plot the distribution of some interactions for different groups. Importantly, we only select events that occur over a time window (here 14 days) that covers the latest activities of each user-phase, to ensure that identified behaviour is linked to churn events.