How to predict customer churn using Machine Learning model on Spark?
By: Monika Bagyal
Tags: Data Science, Spark, Pyspark, Churn Prediction, Machine Learning

How to predict customer churn using Machine Learning model on Spark?

Photo by Oleg Gratilo on Unsplash

If you have worked with Machine Learning models before, you know the feeling when you are dealing with huge datasets. At some point, we all realize this barrier and choose to work with a smaller portion of data to start with data analysis and training models.

Introduction

Big data is very vague, but we all can agree that whenever we feel the restriction of computational power on our laptops or Machines, due to the huge size of a dataset, we see it as Big data problem. And, let’s face it, python data analysis libraries like pandas have their own limitations to manipulate huge datasets due to its local-memory constraints and performance issues.

Business Understanding

Sparkify is a fictional music streaming service like Spotify or Pandora and I will be using Sparkify Churn Prediction as a problem statement to explain this article in detail. I will be using PySpark for this project.

Note: Sparkify dataset was provided by Udacity Data Scientist Nanodegree program and this article is part of my Capstone project.

Data Understanding

I began with Exploratory Data Analysis on my local machine with a small Sparkify dataset provided by Udacity.

# loading dataset

df = spark.read.json("mini_sparkify_event_data.json") # columns in dataset

df.printSchema() root

|-- artist: string (nullable = true)

|-- auth: string (nullable = true)

|-- firstName: string (nullable = true)

|-- gender: string (nullable = true)

|-- itemInSession: long (nullable = true)

|-- lastName: string (nullable = true)

|-- length: double (nullable = true)

|-- level: string (nullable = true)

|-- location: string (nullable = true)

|-- method: string (nullable = true)

|-- page: string (nullable = true)

|-- registration: long (nullable = true)

|-- sessionId: long (nullable = true)

|-- song: string (nullable = true)

|-- status: long (nullable = true)

|-- ts: long (nullable = true)

|-- userAgent: string (nullable = true)

|-- userId: string (nullable = true) # Number of Rows

df.count() 286500 # showing types of events in dataset or pages a user might visit while using the Sparkify app. df.select("page").dropDuplicates().show() +--------------------+

| page|

+--------------------+

| Cancel|

| Submit Downgrade|

| Thumbs Down|

| Home|

| Downgrade|

| Roll Advert|

| Logout|

| Save Settings|

|Cancellation Conf...|

| About|

| Settings|

| Add to Playlist|

| Add Friend|

| NextSong|

| Thumbs Up|

| Help|

| Upgrade|

| Error|

| Submit Upgrade|

+--------------------+

Data Preparation

During data preparation, I dealt with data challenges like missing values, data cleaning, imputing categorical variables and identifying a target variable to predict churned customers.

To accurately predict churned customers, we need their userId information, therefore cleaning data by removing all rows where userId is either null or empty.

Adding, a target variable as Churn. If a user has visited the “Cancellation Confirmation” page, then the Churn column has a value 1 else 0.

Data Visualizations

Let’s have a deeper understanding of Sparkify customers, using visualizations.