Sparkify: churn prediction with pySpark
By: Juan Pablo Escobedo
Tags: Spark, Udacity, Machine Learning, Churn Prediction, Pyspark

Sparkify is a fictious music service where users can stream music.

Users can stream music for free or paid for a premium subscription. At any time an user can change their subscription plan or cancel the service.

Our goal is to discover users that are about to cancel their account, so the company can offer them an incentive to keep them on board. Some of the incentives are trial subscriptions for free users, discounts, or even give them a social incentive like showing them what their friends are listening.

The code used to write this article can be found here.

Dataset Description

Everytime an user interacts with the service, the system generates events which are stored for later analysis. Some of the events in the dataset are: user subscription plan upgrade/downgrade, playing a new song, or cancelling an account.

Schema

Each event contains the following information

User Information

User id

First name

Last name

Gender

Location

Current state

Session id

Auth: whether the user is logged in or not

Level: free or paid subscription

Artist: the last played artist

Song: last played song

Event information

Page: call that triggered the event

Method: http method used to call the page

Status: response http code when calling the page

User agent: browser used to call the page

Ts: timestamp of the event

Data Exploration

Exploration and data manipulation will be performed with pySpark and a small sample of the full dataset.

There are 286,500 events and 225 users in the sample dataset. Since we are only interested on the users that cancel their accounts, we will ignore the events without an user id.

Single sample

[Row(artist='Martha Tilston', auth='Logged In', firstName='Colin', gender='M', itemInSession=50, lastName='Freeman', length=277.89016, level='paid', location='Bakersfield, CA', method='PUT', page='NextSong', registration=1538173362000, sessionId=29, song='Rockpools', status=200, ts=1538352117000, userAgent='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0', userId='30')]

Authentication states and subscription levels

Both subscription levels cancel their accounts

+---------+-----+

| auth|level|

+---------+-----+

|Cancelled| paid|

|Cancelled| free|

|Logged In| free|

|Logged In| paid|

+---------+-----+

Subscription level of users that cancelled their account

+-----+-----+

|level|count|

+-----+-----+

| free| 21|

| paid| 31|

+-----+-----+

Pages

+--------------------+

| page|

+--------------------+

| About|

| Add Friend|

| Add to Playlist|

| Cancel|

|Cancellation Conf...|

| Downgrade|

| Error|

| Help|

| Home|

| Logout|

| NextSong|

| Roll Advert|

| Save Settings|

| Settings|

| Submit Downgrade|

| Submit Upgrade|

| Thumbs Down|

| Thumbs Up|

| Upgrade|

+--------------------+

Pages, methods and status

Downgrade page doesn’t downgrade the account until Submit Downgrade is actually visited. The same goes for Upgrade and Submit Upgrade when upgrading and account, or Cancellation Confirmation and Cancel when cancelling an account.

Note the 307 code, wich is defined as a Temporary Redirect, meaning that probably when the user hits the confirmation button, she gets redirected to another page. While the 200 and GET is simply a request for information (display a webpage).

+------+------+--------------------+

|status|method| page|

+------+------+--------------------+

| 200| GET| About|

| 307| PUT| Add Friend|

| 200| PUT| Add to Playlist|

| 307| PUT| Cancel|

| 200| GET|Cancellation Conf...|

| 200| GET| Downgrade|

| 404| GET| Error|

| 200| GET| Help|

| 200| GET| Home|

| 307| PUT| Logout|

| 200| PUT| NextSong|

| 200| GET| Roll Advert|

| 307| PUT| Save Settings|

| 200| GET| Settings|

| 307| PUT| Submit Downgrade|

| 307| PUT| Submit Upgrade|

| 307| PUT| Thumbs Down|

| 307| PUT| Thumbs Up|

| 200| GET| Upgrade|

+------+------+--------------------+

Sample of a cancellation sequence (from bottom to top)

[Row(userId='100014', ts=1542740649000, page='Cancellation Confirmation', level='paid', phase=0),

Row(userId='100014', ts=1542740642000, page='Cancel', level='paid', phase=0),

Row(userId='100014', ts=1542740641000, page='Downgrade', level='paid', phase=1),

Row(userId='100014', ts=1542740628000, page='NextSong', level='paid', phase=1),

Row(userId='100014', ts=1542740352000, page='NextSong', level='paid', phase=1)]

Pages visited by users before cancelling

From the next graph we can see some users:

Thumbds Down, meaning they are not happy with the music.

meaning they are not happy with the music. Downgrade , they go to the downgrade page without actually downgrading. They might be looking for the cancel page, or when not been able to downgrade they just cancel their account.

, they go to the downgrade page without actually downgrading. They might be looking for the cancel page, or when not been able to downgrade they just cancel their account. Roll Advert, see an ad

see an ad Settings, looking for the cancel procedure

Two actions look interesting: Downgrade and Roll Advert.

246 times a premium user got an ad. Is this a normal feature or a bug?

Pages visited by users that cancelled vs users that didn’t

Both groups have similar behavior.

But as we can see, there are a lot more Downgrade page visits than actual submissions, contrary to the cancellation submissions. Something is preventing the users to downgrade, whether is a technical error, an UX flawn, or a simple user psychological behavior, it should be investigated further.

Data Preparation

Next, we will prepare the dataset used to train our machine learning model. For that, we will take the data as user features instead of a timeseries, meaning there is going to be only one row for each user.

We add a new column called churn, this column is going to be our target label. It has a 1 when a user has cancelled her account or a 0 if she hasn’t.

root

|-- userId: string (nullable = true)

|-- page: string (nullable = true)

|-- churn: integer (nullable = false)

2. For each page in the log data, we add a column. The values on these colums are the times a user visited each page.

root

|-- userId: string (nullable = true)

|-- About: integer (nullable = true)

|-- Add Friend: integer (nullable = true)

|-- Add to Playlist: integer (nullable = true)

|-- Downgrade: integer (nullable = true)

|-- Error: integer (nullable = true)

|-- Help: integer (nullable = true)

|-- Home: integer (nullable = true)

|-- Logout: integer (nullable = true)

|-- NextSong: integer (nullable = true)

|-- Roll Advert: integer (nullable = true)

|-- Save Settings: integer (nullable = true)

|-- Settings: integer (nullable = true)

|-- Submit Downgrade: integer (nullable = true)

|-- Submit Upgrade: integer (nullable = true)

|-- Thumbs Down: integer (nullable = true)

|-- Thumbs Up: integer (nullable = true)

|-- Upgrade: integer (nullable = true)

3. We then add columns called last_page_<page> referring to the las page visited by the users. 1 if the page was the last one visited, 0 if it wasn’t.

root

|-- userId: string (nullable = true)

|-- last_page_About: integer (nullable = true)

|-- last_page_Add Friend: integer (nullable = true)

|-- last_page_Add to Playlist: integer (nullable = true)

|-- last_page_Downgrade: integer (nullable = true)

|-- last_page_Help: integer (nullable = true)

|-- last_page_Home: integer (nullable = true)

|-- last_page_Logout: integer (nullable = true)

|-- last_page_NextSong: integer (nullable = true)

|-- last_page_Roll Advert: integer (nullable = true)

|-- last_page_Settings: integer (nullable = true)

|-- last_page_Thumbs Down: integer (nullable = true)

|-- last_page_Thumbs Up: integer (nullable = true)

4. We do the same for the last level the user had

root

|-- userId: string (nullable = true)

|-- last_level_free: integer (nullable = true)

|-- last_level_paid: integer (nullable = true)

The final dataset looks like this

Churn Prediction

Finally we are going to create a machine learning model, train it and measure its accuracy to find users that could churn.

Prepare features

In Spark, we need to create a new column of type SparseVector with all the features used in the model. For that we use VectorAssembler with all the columns defined before.

A single row looks something like this, note that when a feature has a value of 0 it doesn’t show in the vector:

[Row(userId='100010', level='free', churn=0, features=SparseVector(31, {0: 1.0, 1: 4.0, 2: 7.0, 5: 2.0, 6: 11.0, 7: 5.0, 8: 275.0, 9: 52.0, 14: 17.0, 15: 5.0, 16: 2.0, 23: 1.0, 29: 1.0}))]

2. Create a decission tree classifier

Then we split our dataset by 70% for training and 30% for testing. And create a DecissionTreeClassifier.

3. Hyperparameter tuning

To try several parameters of the classifier we use a CrossValidator. The parameters tested are:

maxDepth: [5, 10, 30]

maxBins: [32, 64]

4. Predictions

After fiting the model, maxDepth = 5 and maxBins = 32 are the chosen parameters.

+----------+-----+

|prediction|label|

+----------+-----+

| 0.0| 0|

| 1.0| 1|

| 0.0| 0|

| 1.0| 1|

| 0.0| 0|

+----------+-----+

5. Evaluation

Finally we evaluate the model using a MulticlassClassificationEvaluator throwing the next results.

Accuracy = 0.919355

F1 score = 0.918488

Weighted Precision = 0.918234

Weighted Recall = 0.919355

F1 score is the value that we should aim to improve, since we are dealing with an unbalanced dataset. Meaning we have just a few churn users.

Conclussion

With the help of this machine learning model, Sparkify team could easily predict the churn classification each time an event is trigered.

When a user is about to cancel her account, the system could automatically offer an incentive to keep the user engaged. We could offer users free months, recommend new music, or bring up their favorite songs.

Challenges

From previous experience working with spark, keeping the system optimal is a challenge. From a code perspective, one must pay special attention to data shuffle, the process of moving data from one node to the other, extracting the data in an optimal way, caching data.

From a infrasture management perspective, one must know how the JVM manages memory to optimize its parameters, specially on huge datasets. Use the rigth amount of hardware to finish the task in an apropiate time length.

Improvement

The model results can be improved using other machine learning algorithms. A random forest could throw better results at a highter cost, and a deep learning model probably would perform even better.

In this model we are dealing with the user information as a single static data point. We are not considering the sequence of the user actions (except for the last action performed), and all the information those events have.

For example, an user could be more prone to cancel her account at lunch time, or after receiving a 404 http error, three times one after another, when trying to play her favorite song.

To deal with timeseries data, spark has a streaming implementation. And, another machine learning algorithm should be used, like SARIMA or deep learnig LSTM.