Customer Churn Prediction with PySpark on IBM Watson Studio, AWS and Databricks
By: Josh Xin Jie Lee
Tags: Data Science, Predictive Analytics, Data Analytics, Machine Learning, Big Data

Dataset

The schema of the dataset:

A row from the dataset:

One of the key column to look out for in this dataset is the ‘page’ column. The ‘page’ column keeps track of what pages a user visits.

This is a list of the pages that users can visit:

Cancel: User has visited the cancel page. Does not mean cancellation was completed. Submit Downgrade: User has submitted a downgrade from premium to free tier Thumbs Down: User gave a thumbs down. Home: User visited the home page Downgrade: User visited the downgrade page. Does not mean downgrade is submitted. Roll Advert: Advertisements are played. Logout: User logged out. Save Settings: User made some changes to settings and save them. Cancellation Confirmation: User cancelled the subscription. About: User visited the about page. Submit Registration: User submitted a registration request. Settings: User visited the settings page. Login: User logged in. Register: User visited the registration page. Does not mean registration was completed. Add to Playlist: User added songs to playlist. Add Friend: User added a friend. Next Song: User listened to a song. Thumbs Up: User gave a thumbs up. Help: User visited the help page. Upgrade: User upgraded from free to premium tier.

Of these, the pages which we should pay attention to are perhaps: ‘Next Song’ which tracks the songs played by a user, ‘Submit Downgrade’ which tracks when a user submits a downgrade request and ‘Cancellation Confirmation’ which tracks when a user’s cancellation request is confirmed.

We will know when a customer churn by searching for a ‘Submit Downgrade’ or ‘Cancellation Confirmation’ page.

Note that a user can visit the ‘Downgrade’ and ‘Cancel’ page but not submit a downgrade or cancellation request.

The rest of the pages are relatively straightforward. They indicate that the user has visited the relevant page.

As mentioned earlier, there are 2 months of data available.

Data Preprocessing

Reading in the data is slightly different depending on the cloud computing service you are using. Each service’s method of reading in the dataset are shared above in the setup instructions.

If you are reading in the data on your local computer, this is the code for the small dataset:

# create a Spark session

spark = SparkSession.builder \

.master("local") \

.appName("Sparkify") \

.getOrCreate() df = spark.read.json("mini_sparkify_event_data.json")

This code assumes that the dataset is located in the same file directory as your code. You might need to change the path of the dataset depending on where you store your dataset.

Data Cleaning

There are a couple of columns with null values. Notably, entries missing firstName , gender , lastName , location , registration and userAgent belong to users who did not log in or did not register.

Number of null values for each feature on the full dataset

Since we do not know which users these entries are tracking and most of these entries are associated with the login or home page, these entries provide little value. Hence, we can discard them from our analysis.

# filter out all entries with missing names.

# these also removes entries with missing gender, lastName,

# location, registration and userAgent df = df.filter(df.firstName.isNotNull())

Pages that are not ‘NextSong’ have null values for artist , length and sessionId . All of these variables are only valid when songs are played (they are only non-null when page =’NextSong’), hence we can leave them unmodified.

Feature Engineering

First, we transform the timestamps to datetime. The original timestamps are in milliseconds, so we have to divide them by 1000 before converting them.

# original timestamp in milliseconds, so divide by 1000 adjust_timestamp = udf(lambda x : x//1000, IntegerType())

df = df.withColumn("ts_adj", adjust_timestamp('ts')) # convert adjusted timestamp to datetime

df = df.withColumn("datetime", from_unixtime(col("ts_adj"))) # convert registration timestamp to datetime

df = df.withColumn("reg_adj", adjust_timestamp('registration')) # convert adjusted registration timestamp to datetime

df = df.withColumn("reg_datetime", from_unixtime(col("reg_adj"))) # drop all the timestamp columns. Will not need them

columns_to_drop = ['registration', 'ts', 'ts_adj', 'reg_adj']

df = df.drop(*columns_to_drop)

Next, we can label the months from 0 to N where N represents the total number of months available in the dataset. We can approximate the start date of the analysis as ‘2018–10–01 00:00:00’.

# add start date of analysis

df = df.withColumn('analysis_start_date',\

lit('2018-10-01 00:00:00')) # number the months starting from the very first month of the # analysis

df = df.withColumn("month_num",\

floor(months_between(df.datetime,\

df.analysis_start_date)))

Replace the string variables for gender and level with binary (0 or 1) variables.

# engineer free or paid binary variable

# free: 0, paid: 1

df = df.replace(["free", "paid"], ["0", "1"], "level")



# engineer male and female binary binary variable

# male: 0, female: 1

df = df.replace(["M", "F"], ["0", "1"], "gender")

We can define churn events as whenever users visit a ‘Cancellation Confirmation’ or ‘Submit Downgrade’ pages.

def define_churn(x):

"""

Defining churn as cancellation of service or downgrading from premium to free tier.

"""

if x == "Cancellation Confirmation":

return 1

elif x == "Submit Downgrade":

return 1

else:

return 0 churn_event = udf(lambda x : define_churn(x), IntegerType())

df = df.withColumn("churn", churn_event("page"))

Generating monthly statistics for each user:

# number of register page visits

df_register = df.select('userId', 'month_num', 'page') \

.where(df.page=="Register") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numRegister') # number of cancel page visits

df_cancel = df.select('userId', 'month_num', 'page') \

.where(df.page=="Cancel") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numCancelVisits') # number of upgrade page visits

df_upgrade = df.select('userId', 'month_num', 'page') \

.where(df.page=="Upgrade") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numUpgradeVisits') # number of downgrade page visits

df_downgrade = df.select('userId', 'month_num', 'page') \

.where(df.page=="Downgrade") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numDowngradeVisits') # number of home page visits

df_home = df.select('userId', 'month_num', 'page') \

.where(df.page=="Home") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numHomeVisits') # number of about page visits

df_about = df.select('userId', 'month_num', 'page') \

.where(df.page=="About") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numAboutVisits') # number of setting page visits

df_settings = df.select('userId', 'month_num', 'page') \

.where(df.page=="Settings") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numSettingsVisits') # number of times user save settings changes

df_saveSettings = df.select('userId', 'month_num', 'page') \

.where(df.page=="Save Settings") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numSaveSettings') # number of login page visits

df_login = df.select('userId', 'month_num', 'page') \

.where(df.page=="Login") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numLogins') # number of logout page visits

df_logout = df.select('userId', 'month_num', 'page') \

.where(df.page=="Logout") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numLogouts') # number of songs added to playlist

df_addPlaylist = df.select('userId', 'month_num', 'page') \

.where(df.page=="Add to Playlist") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numAddPlaylists') # number of friends added

df_addFriend = df.select('userId', 'month_num', 'page') \

.where(df.page=="Add Friend") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numFriends') # number of thumbs up given

df_thumbsUp = df.select('userId', 'month_num', 'page') \

.where(df.page=="Thumbs Up") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numThumbsUp') # number of thumbs down given

df_thumbsDown = df.select('userId', 'month_num', 'page') \

.where(df.page=="Thumbs Down") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numThumbsDown') # number of advertisements rolled

df_advert = df.select('userId', 'month_num', 'page') \

.where(df.page=="Roll Advert") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numAdverts') # number of songs played

df_songsPlayed = df.select('userId', 'month_num', 'page') \

.where(df.page=="NextSong") \

.groupBy('userId', 'month_num') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'numSongsPlayed') # total amount of time user listened to songs

df_totalListen = df.select('userId', 'month_num', 'length') \

.groupBy('userId', 'month_num') \

.agg({'length':'sum'}) \

.withColumnRenamed('sum(length)', 'totalListenTime')



# number of songs played per session

df_songsPerSession = df.select('userId', 'month_num', 'page', 'sessionId') \

.where(df.page=="NextSong") \

.groupBy('userId', 'month_num', 'sessionId') \

.agg({'page':'count'}) \

.withColumnRenamed('count(page)', 'SongsPerSession') # average number of songs played per session

df_avgSongsPerSession = df_songsPerSession.groupBy('userId', 'month_num') \

.agg(avg(df_songsPerSession.SongsPerSession).alias ('avgSongsPerSession'))



# number of singers listened per month

df_singersPlayed = df.select('userId', 'month_num', 'page', 'artist') \

.where(df.page=="NextSong") \

.groupBy('userId', 'month_num') \

.agg(countDistinct(df.artist).alias('numSingersPlayed'))



# number of singers per session

df_singersPerSession = df.select('userId', 'month_num', 'page', 'artist', 'sessionId') \

.where(df.page=="NextSong") \

.groupBy('userId', 'month_num', 'sessionId') \

.agg(countDistinct(df.artist).alias('SingersPerSession')) # average number of singers per session

df_avgSingersPerSession = df_singersPerSession.groupBy('userId', 'month_num') \

.agg(avg(df_singersPerSession.SingersPerSession).alias ('avgSingersPerSession'))



# amount of time spent for each session

df_userSession = df.groupBy("userId", "month_num", "sessionId") \

.agg(((max(unix_timestamp(df.datetime))-min(unix_timestamp (df.datetime)))/60.0).alias('sessionTimeMins')) # average time per session

df_avgUserSession = df_userSession.groupBy('userId', 'month_num').agg(avg(df_userSession.sessionTimeMins).alias ('avgSessionMins'))



# number of sessions per month

df_numSession = df.select('userId', 'month_num', 'sessionId').dropDuplicates() \

.groupby('userId', 'month_num').agg({'sessionId':'count'}) \

.withColumnRenamed('count(sessionId)', 'numSessions') # if user had premium level this month

# if user had premium at any point of the month, assumer he/she has # premium for the whole month for simplicity

df_level = df.select('userId', 'month_num', 'level') \

.groupBy('userId', 'month_num') \

.agg({'level':'max'}) \

.withColumnRenamed('max(level)', 'level') # find user's gender

# assuming nobody changes gender midway

df_gender = df.select('userId', 'month_num', 'gender') \

.groupBy('userId', 'month_num') \

.agg({'gender':'max'}) \

.withColumnRenamed('max(gender)', 'gender')



# start of each month

df = df.withColumn("start_of_month", expr("add_months (analysis_start_date, month_num)"))

# days since registration to start of each month

df = df.withColumn("daysSinceReg", datediff(df.start_of_month, df.reg_datetime))

df_daysReg = df.select('userId', 'month_num', 'daysSinceReg') \

.groupBy('userId', 'month_num') \

.agg(min(df.daysSinceReg).alias('daysSinceReg')) # did user churn this month

df_churn = df.select('userId', 'month_num', 'churn') \

.groupBy('userId', 'month_num') \

.agg({'churn':'max'}) \

.withColumnRenamed('max(churn)', 'churn')

Concatenating these monthly statistics into a new data-frame:

all_data = df_register.join(df_cancel, ['userId', 'month_num'], 'outer') \

.join(df_upgrade, ['userId', 'month_num'], 'outer') \

.join(df_downgrade, ['userId', 'month_num'], 'outer') \

.join(df_home, ['userId', 'month_num'], 'outer') \

.join(df_about, ['userId', 'month_num'], 'outer') \

.join(df_settings, ['userId', 'month_num'], 'outer') \

.join(df_saveSettings, ['userId', 'month_num'], 'outer') \

.join(df_login, ['userId', 'month_num'], 'outer') \

.join(df_logout, ['userId', 'month_num'], 'outer') \

.join(df_addPlaylist, ['userId', 'month_num'], 'outer') \

.join(df_addFriend, ['userId', 'month_num'], 'outer') \

.join(df_thumbsUp, ['userId', 'month_num'], 'outer') \

.join(df_thumbsDown, ['userId', 'month_num'], 'outer') \

.join(df_advert, ['userId', 'month_num'], 'outer') \

.join(df_songsPlayed, ['userId', 'month_num'], 'outer') \

.join(df_totalListen, ['userId', 'month_num'], 'outer') \

.join(df_avgSongsPerSession, ['userId', 'month_num'], 'outer') \

.join(df_singersPlayed, ['userId', 'month_num']) \

.join(df_avgSingersPerSession, ['userId', 'month_num'], 'outer') \

.join(df_avgUserSession, ['userId', 'month_num'], 'outer') \

.join(df_numSession, ['userId', 'month_num'], 'outer') \

.join(df_level, ['userId', 'month_num'], 'outer') \

.join(df_gender, ['userId', 'month_num'], 'outer') \

.join(df_daysReg, ['userId', 'month_num'], 'outer') \

.join(df_churn, ['userId', 'month_num'], 'outer')

Next, generate 1-month lagged features. These will be used as input features for the model, rather than the current month statistics, since we do not want any look-ahead biases.

windowlag = (Window.partitionBy('userId').orderBy('month_num'))



# generate 1 month lag features

all_data = all_data.withColumn('numRegister_lastMonth', lag(all_data ['numRegister']).over(windowlag)) all_data = all_data.withColumn('numCancelVisits_lastMonth', lag (all_data['numCancelVisits']).over(windowlag)) all_data = all_data.withColumn('numUpgradeVisits_lastMonth', lag (all_data['numUpgradeVisits']).over(windowlag)) all_data = all_data.withColumn('numDowngradeVisits_lastMonth', lag (all_data['numDowngradeVisits']).over(windowlag)) all_data = all_data.withColumn('numHomeVisits_lastMonth', lag (all_data['numHomeVisits']).over(windowlag)) all_data = all_data.withColumn('numAboutVisits_lastMonth', lag (all_data['numAboutVisits']).over(windowlag)) all_data = all_data.withColumn('numSettingsVisits_lastMonth', lag (all_data['numSettingsVisits']).over(windowlag)) all_data = all_data.withColumn('numSaveSettings_lastMonth', lag (all_data['numSaveSettings']).over(windowlag)) all_data = all_data.withColumn('numLogins_lastMonth', lag(all_data ['numLogins']).over(windowlag)) all_data = all_data.withColumn('numLogouts_lastMonth', lag(all_data ['numLogouts']).over(windowlag)) all_data = all_data.withColumn('numAddPlaylists_lastMonth', lag (all_data['numAddPlaylists']).over(windowlag)) all_data = all_data.withColumn('numFriends_lastMonth', lag(all_data ['numFriends']).over(windowlag)) all_data = all_data.withColumn('numThumbsUp_lastMonth', lag(all_data ['numThumbsUp']).over(windowlag)) all_data = all_data.withColumn('numThumbsDown_lastMonth', lag (all_data['numThumbsDown']).over(windowlag)) all_data = all_data.withColumn('numAdverts_lastMonth', lag(all_data ['numAdverts']).over(windowlag)) all_data = all_data.withColumn('numSongsPlayed_lastMonth', lag (all_data['numSongsPlayed']).over(windowlag)) all_data = all_data.withColumn('totalListenTime_lastMonth', lag (all_data['totalListenTime']).over(windowlag)) all_data = all_data.withColumn('avgSongsPerSession_lastMonth', lag (all_data['avgSongsPerSession']).over(windowlag)) all_data = all_data.withColumn('numSingersPlayed_lastMonth', lag (all_data['numSingersPlayed']).over(windowlag)) all_data = all_data.withColumn('avgSingersPerSession_lastMonth', lag (all_data['avgSingersPerSession']).over(windowlag)) all_data = all_data.withColumn('avgSessionMins_lastMonth', lag (all_data['avgSessionMins']).over(windowlag)) all_data = all_data.withColumn('numSessions_lastMonth', lag(all_data ['numSessions']).over(windowlag)) all_data = all_data.withColumn('level_lastMonth', lag(all_data ['level']).over(windowlag))

The generated features that will be used for our predictive model are:

numRegister_lastMonth : Number of times user registered last month numCancelVisits_lastMonth : Number of times user visited cancellation page last month numUpgradeVisits_lastMonth : Number of times user visited upgrade page last month numDowngradeVisits_lastMonth : Number of times user visited downgrade page last month numHomeVisits_lastMonth : Number of times user visited home page last month numAboutVisits_lastMonth : Number of times user visited about page last month numSettingsVisits_lastMonth : Number of times user visited settings page last month numSaveSettings_lastMonth : Number of times user saved changes to settings last month numLogins_lastMonth : Number of times user login last month numLogouts_lastMonth : Number of times user logout last month numAddPlaylists_lastMonth : Number of songs user added to playlist last month numFriends_lastMonth : Number of friends user added last month numThumbsUp_lastMonth : Number of thumbs up user gave last month numThumbsDown_lastMonth : Number of thumbs down user gave last month numAdverts_lastMonth : Number of advertisements played to user last month numSongsPlayed_lastMonth : Number of songs user played last month totalListenTime_lastMonth : Total listening time for user last month avgSongsPerSession_lastMonth : Average number of songs played per session by user last month numSongsPlayed_lastMonth : Number of songs played by user last month avgSingersPerSession_lastMonth : Average number of singers played per session by user last month avgSessionMins_lastMonth : Average number of minutes per session by user last month numSessions_lastMonth : Number of sessions by user last month daysSinceReg : Number of days since registration to the first day of the current month for each user level_lastMonth : Tracks if a user was a paid or free user last month. If user was a paid user at any period of time last month, we will assume he/she was a paid user for the entire month for simplicity

All missing values are imputed with 0 since missing values generally indicate no page visits, no songs played, etc.

Exploratory Data Analysis

The following exploratory data analysis was obtained from the medium-sized instance of the dataset. Similar trends should be observed in both the small and large instances of the dataset.

Shown below are the boxplots of various user statistics from the first month grouped by whether these users churned during the second month.

We are interested to see if there are any discernable patterns in past user behavior that could indicate whether a user will churn in the near future.

A surprising observation is that users who churned in the second month seem to be far more active on the digital music service during their first month, as compared to users who did not churn. Churned customers consistently have higher page visits, played more songs, listened to more singers, gave more thumbs up and thumbs down, etc.

Perhaps, this suggests that users who are likely to churn next month will often try to make full use of their service in the current month.

In addition, we note that the number of registrations, the number of visits to the cancel page and the number of logins were missing.

When a user registers or login, the system does not know the user’s id. Hence no information on these activities were recorded for the customers.

As for the missing statistic regarding page visits to the cancel page, I note that none of the users staying with the service in the second month visited the cancel page during the first month. It is likely that any user who visited the cancel page during the first month probably churned during the month, hence none of them were with the service during the second month.

Premium users are far more likely to churn

Premium users seem far more likely to churn as compared to free users. Note that churn in this case could mean either downgrading from premium to free tier or cancelling their account altogether.

Male users are slightly less likely to churn

In addition, it seems that male users are slightly less likely to churn as compared to female users.

During the second month, the number of users who churned vs number of users who did not churn

The number of users who churned in the second month are far lower than the number of users who did not churn. Hence, we will need to use the appropriate metrics to evaluate our model to ensure that our results are not deceiving. We could also employ techniques to handle imbalanced datasets, such as SMOTE, to improve our model’s ability to predict users who will churn.

In conclusion, it is likely that most churn resulted from users downgrading from premium to free tier. I hypothesized that there was a lower introductory premium pricing plan for new premium users that did not require any long-term commitment. Many of these new premium users were likely unwilling to make a long-term commitment to the music service, and wanted to maximize their usage of the service during the month when they had premium service.

Alternatively, it could also be that the premium users who churned found the service unsatisfactory after prolonged usage, and thus wanted to downgrade their subscription.

Modelling

From the exploratory data analysis, we note that past customer activities can provide a relatively good indication of whether they will churn in the near future. We will now build predictive models, with the aid of Spark ML library, to predict when these customers will potentially churn.

Spark ML library requires input features to be numeric, so we have to convert all string variables to integers or floats.

# convert userId, gender, level, level_lastMonth to numeric

convert_numeric = ['userId', 'level', 'gender', 'level_lastMonth']

for feat in convert_numeric:

featName = feat + "_n"

all_data = all_data.withColumn(featName, all_data[feat].cast ("float"))

all_data = all_data.drop(feat)

You can check the datatypes of the features by running all_data.persist() . In addition, ensure that there are no null or ‘NaN’ values in your input features.

Next, we drop entries from month 0, since we do not have previous months’ statistics to work with. We will drop the current month statistics and keep last month’s lagged features and the churn labels.

# first month is month 0

model_data = all_data \

.filter(all_data.month_num>0) \

.select('userId_n', 'month_num',\ 'numUpgradeVisits_lastMonth',\

'numDowngradeVisits_lastMonth',\

'numHomeVisits_lastMonth',\

'numAboutVisits_lastMonth',\

'numSettingsVisits_lastMonth',\

'numSaveSettings_lastMonth',\

'numLogouts_lastMonth',\

'numAddPlaylists_lastMonth',\

'numFriends_lastMonth',\

'numThumbsUp_lastMonth',\

'numThumbsDown_lastMonth',\

'numAdverts_lastMonth',\

'numSongsPlayed_lastMonth',\

'totalListenTime_lastMonth',\

'avgSongsPerSession_lastMonth',\

'numSingersPlayed_lastMonth',\

'avgSingersPerSession_lastMonth',\

'avgSessionMins_lastMonth',\

'numSessions_lastMonth',\

'level_lastMonth_n',\

'gender_n', 'daysSinceReg', 'churn' ).withColumnRenamed('churn', 'label')

We will store the input features and the labels in a data frame model_data .

As a recap, we will be predicting customer churn during the second month using their activity data from the first month. Hence, we will perform a simple train-test split on the second month to obtain the training and test datasets.

train,test = model_data.randomSplit([0.8, 0.2], seed=50)

Note that all input features for a data point have to be placed in a single vector called features . It is recommended to scale the input features as well. The column churn should be renamed as label . Spark ML library will look for the features and label vectors, so use this naming convention.

inputColumns = ['userId_n', 'month_num',\ 'numUpgradeVisits_lastMonth',\

'numDowngradeVisits_lastMonth',\ 'numHomeVisits_lastMonth',\

'numAboutVisits_lastMonth',\ 'numSettingsVisits_lastMonth',\

'numSaveSettings_lastMonth',\ 'numLogouts_lastMonth',\

'numAddPlaylists_lastMonth',\

'numFriends_lastMonth',\ 'numThumbsUp_lastMonth',\

'numThumbsDown_lastMonth',\ 'numAdverts_lastMonth',\

'numSongsPlayed_lastMonth',\

'totalListenTime_lastMonth',\

'avgSongsPerSession_lastMonth',\

'numSingersPlayed_lastMonth',\

'avgSingersPerSession_lastMonth',\ 'avgSessionMins_lastMonth',\

'numSessions_lastMonth',\

'level_lastMonth_n', 'gender_n',\

'daysSinceReg'] assembler = VectorAssembler(inputCols=inputColumns,\

outputCol="FeaturesVec") scaler = StandardScaler(inputCol="FeaturesVec",\

outputCol="features",\

withMean=True, withStd=True)

We will be trying 3 types of models: logistic regression, linear support vector classifier and gradient-boosted trees (GBT).

# set max_iter to 10 to reduce computation time # Logistic Regression

lr=LogisticRegression(maxIter=10)

pipeline_lr = Pipeline(stages=[assembler, scaler, lr]) # Support Vector Machine Classifier

svc = LinearSVC(maxIter=10)

pipeline_svc = Pipeline(stages=[assembler, scaler, svc]) # Gradient Boosted Trees

gbt = GBTClassifier(maxIter=10, seed=42)

pipeline_gbt = Pipeline(stages=[assembler, scaler, gbt])

The test accuracy, f1-score, precision and recall will be reported. F1-score will be preferred since the labels are imbalanced.

where:

TN (True Negatives): Customers we predict will not churn and who did not churn in reality. FP (False Positives): Customers predicted to churn but did not churn in reality. FN (False Negatives): Customers we predict will not churn but churned in reality. TP (True Positives): Customers predicted to churn and churned in reality.

The precision tells us how precise our predictions are (of those predicted to churn, how many did churn), while recall indicates how well our model recalled (how many of the original churned customer did the model manage to find).

The f1-score can be viewed as the weighted average of precision and recall.

Since the number of customers who churned is smaller than the number of customers who did not churn, f1-score will provide a more accurate representation of the model’s performance, as opposed to accuracy.

A word of caution when using PySpark’s MulticlassClassificationEvaluator to report the f1-score for a binary classification task, is that the in-built function will treat both labels 0 and 1 as separate classes and return the weighted f1-score of both classes. This will produce an overly optimistic score that can be deceiving. Hence, I opted to define my own function to calculate the f1-score. More details can be found in the code.

Results

We will be performing k-fold cross-validation with 5 folds, and the models will be optimized using the area under the PR curve.

Using the area under the PR curve is preferred over using the area under the ROC curve, since the imbalance of classes in the dataset meant that using the area under the ROC curve will lead to an overly optimistic picture.

The following results are obtained from training our model on the full dataset.

Logistic Regression

paramGrid = ParamGridBuilder() \

.addGrid(lr.regParam,[0.0, 0.05, 0.1]) \

.build() cv_lr = CrossValidator(estimator=pipeline_lr,\

estimatorParamMaps=paramGrid,\

evaluator=\

BinaryClassificationEvaluator(metricName=\

"areaUnderPR"),\

numFolds=5, seed=42) cvModel_lr = cv_lr.fit(train) lr_results = cvModel_lr.transform(test) evaluate_model(lr_results)

Accuracy: 0.790473641959

F1-Score: 0.387802971071

Precision: 0.666666666667

Recall: 0.273428886439

Confusion Matrix:

TN:2706.0 | FP:124.0

FN:659.0 | TP: 248.0

Linear Support Vector Classifier

paramGrid = ParamGridBuilder() \

.addGrid(svc.regParam,[0.0, 0.05, 0.1]) \

.build() cv_svc = CrossValidator(estimator=pipeline_svc,\

estimatorParamMaps=paramGrid,\

evaluator=\

BinaryClassificationEvaluator(metricName=\

"areaUnderPR"),\

numFolds=5, seed=42) cvModel_svc = cv_svc.fit(train) svc_results = cvModel_svc.transform(test) evaluate_model(svc_results)

Accuracy: 0.773882793685

F1-Score: 0.189837008629

Precision: 0.727941176471

Recall: 0.109151047409

Confusion Matrix:

TN:2793.0 | FP:37.0

FN:808.0 | TP: 99.0

Gradient-Boosted Trees

paramGrid = ParamGridBuilder() \ .addGrid(gbt.minInstancesPerNode,[5]) \ .addGrid(gbt.maxDepth,[7])\ .addGrid(gbt.subsamplingRate,[0.75])\

.build() cv_gbt = CrossValidator(estimator=pipeline_gbt,\

estimatorParamMaps=paramGrid,\

evaluator=\

BinaryClassificationEvaluator(metricName=\

"areaUnderPR"),\

numFolds=5, seed=42) cvModel_gbt = cv_gbt.fit(train) gbt_results = cvModel_gbt.transform(test) evaluate_model(gbt_results)

Accuracy: 0.776826331282

F1-Score: 0.409348441926

Precision: 0.572277227723

Recall: 0.318632855568

Confusion Matrix: