Customer Churn Prediction within Music Streaming using PySpark | by Isak Kabir | Towards Data Science

It is key for a music streaming business to identify users that are likely to churn, i.e. users who are at risk of downgrading from premium and free subscription to cancelling the service. If a music streaming business accurately identifies such users in advance, they can offer them discounts or other similar incentives and save millions in revenues. It is a well-known fact that it is more expensive to acquire a new customer than it is to retain an existing one. This is because returning customers will likely spend 67% more on your company’s products and services.
We want to identify the users that could potentially cancel their account and leave the service. Our goal in this project was to help a fictional business (similar to Spotify and Pandora) by building and training a binary classifier that is able to accurately identify users who cancelled the music streaming service, based on the patterns obtained from their past activity and interaction with the service.
Define churn variable: 1 — users who cancelled their subscription within the observation period, and 0 — users who kept the service throughout
Due to the size of the data-set, the project was carried out by leveraging Apache Spark distributed cluster-computing framework capabilities, using Python API for Spark, PySpark.
# import librariesfrom pyspark import SparkContext, SparkConffrom pyspark.sql import SparkSessionfrom pyspark.sql import Windowfrom pyspark.sql.functions import udf, col, concat, count, lit, avg, lag, first, last, whenfrom pyspark.sql.functions import min as Fmin, max as Fmax, sum as Fsum, round as Froundfrom pyspark.sql.types import IntegerType, DateType, TimestampTypefrom pyspark.ml import Pipelinefrom pyspark.ml.feature import VectorAssembler, Normalizer, StandardScalerfrom pyspark.ml.regression import LinearRegressionfrom pyspark.ml.classification import LogisticRegression, RandomForestClassifier, GBTClassifierfrom pyspark.ml.clustering import KMeansfrom pyspark.ml.tuning import CrossValidator, ParamGridBuilderfrom pyspark.ml.evaluation import BinaryClassificationEvaluator,# create a Spark sessionspark = SparkSession \.builder \.appName(‘CustomerChurn’) \.getOrCreate()# Check Spark configurationspark.sparkContext.getConf().getAll()path = "mini_sparkify_event_data.json"df = spark.read.json(path)
The data-set contains user activity logs recorded between October 1, 2018 and December 01, 2018. The full data-set consists of approximately 26 million rows/logs, whereas the subset contains 286 500 rows. The full data-set collects logs of 22 277 different users, whereas the subset only covers the activities of 225 users. The subset dataset contains 58 300 Free users and 228 000 Paid users. Both data-sets have 18 columns, as listed below.
root |-- artist: string (nullable = true) |-- auth: string (nullable = true) |-- firstName: string (nullable = true) |-- gender: string (nullable = true) |-- itemInSession: long (nullable = true) |-- lastName: string (nullable = true) |-- length: double (nullable = true) |-- level: string (nullable = true) |-- location: string (nullable = true) |-- method: string (nullable = true) |-- page: string (nullable = true) |-- registration: long (nullable = true) |-- sessionId: long (nullable = true) |-- song: string (nullable = true) |-- status: long (nullable = true) |-- ts: long (nullable = true) |-- userAgent: string (nullable = true) |-- userId: string (nullable = true)
Every activity log belongs to a specific user. Seven columns in the data-set represent static user-level information (fixed for all logs belonging to a specific user):
artist: artist that the user was listening touserId: user identifier;sessionId: a unique ID that identifies a single continuous period of use by a user of the service. Multiple users can have sessions labelled with the same sessionIdfirstName: user’s first namelastName: user’s last namegender: user’s gender; 2 categories (M and F)location: user’s locationuserAgent: agent used by the user to access the streaming service; 57 different categoriesregistration: user’s registration timestamplevel (non-static): subscription level; 2 categories (free and paid)page: what page was the user visiting when this event got generated. The different types of pages are detailed in the section below
The page column contains logs for all the page a user has visited in the app.
>>> df.select('page').distinct().show(10)+--------------------+|                page|+--------------------+|              Cancel||    Submit Downgrade||         Thumbs Down||                Home||           Downgrade||         Roll Advert||              Logout||       Save Settings||Cancellation Conf...||               About|+--------------------
Based on the analysis performed it seems that the maximum time between two consecutive logs that still belong to the same session is one hour.
# Explore the auth columndf.groupby('auth').count().show()+----------+------+|      auth| count|+----------+------+|Logged Out|  8249|| Cancelled|    52||     Guest|    97|| Logged In|278102|+----------+------+
We can also see that users are quite active, one of the top users has listed to about 8000 songs in total. The graphs below indicates that churned users are usually from California and New Jersey, mostly paid subscribers are leaving the Music app and more men than women tend to cancel their subscription. California and New York states tend to be denser in population, and therefore could expect higher numbers of churn and higher in overall engagement. It is easy to see from the plot below that the provided Sparkify data-set is a case of an imbalanced dataset, as the share of churned users is only a bit more than 20% (52) compared to 174.
First, we had to transform the original data-set (one row per log) to a data-set with user-level information or statistics (one row per user). We achieved this by performing several mapping (e.g. obtaining user’s gender, length of observation period, etc.) and aggregation steps.
For the few users who registered after October 1 the registration times are not consistent with the actual log timestamps and activity types. We thus had to identify late registrations by finding Submit Registration logs in the page column. This step was not trivial as such log events are not mapped to any userId so those had to be extracted from the sessionId information.
For the few users who registered late the observation start was then set to the timestamp of their first log, whereas for all other users the default October 1 was used.
# Lag the page columnwindowsession = Window.partitionBy('sessionId').orderBy('ts')df = df.withColumn("lagged_page", lag(df.page).over(windowsession))windowuser = Window.partitionBy('userId').orderBy('ts').rangeBetween(Window.unboundedPreceding, Window.unboundedFollowing)# Identify users that registered after the start of observation, and infer the start date accordinglydf = df.withColumn("beforefirstlog", first(col('lagged_page')).over(windowuser))df = df.withColumn("firstlogtime", first(col('ts')).over(windowuser))df = df.withColumn("obsstart",                    when(df.beforefirstlog == "Submit Registration", df.firstlogtime).otherwise(obs_start_default))# For each log compute the time from the beginning of observation...df = df.withColumn("timefromstart", col('ts')-col("obsstart"))# ...and time before the end of observationdf = df.withColumn("timebeforeend", col('obsend')-col('ts'))
Similar to above, there are users who cancelled their service before the end of the default observation period, the so-called churned users. For each such user the end of the respective observation period has been set to the timestamp of his/her last log entry, whereas for all other users the default 1st of December.
The newly created user-level data-set includes the following columns:
lastlevel: user’s last subscription level, transformed into binary format (1 — paid tier, 0 — free tier)gender: gender, transformed into binary format (1 — female, 0 — male)obsstart, obsend: start and end of user-specific observation period
endstate: user’s last interaction in the observation periodnact: user’s total number of interactions in the observation periodnsongs, ntbup, ntbdown, nfriend, nplaylist, ndgrade, nupgrade, nhome, nadvert, nhelp, nsettings, nerror: number of songs played, thumbs up given, thumbs down given, friends added, songs added to playlist, downgrades, upgrades, home page visits, advertisements played, help page visits, settings visits, errors, respectively
nact_recent, nact_oldest: user’s activity in the last and first k days of the observation window, respectivelynsongs_recent, nsongs_oldest: songs played in the last and first k days of the observation window, respectively
# Aggregation by userIddf_user = df.groupby(‘userId’)\.agg( # User-level features first(when(col(‘lastlevel’) == ‘paid’, 1).otherwise(0)).alias(‘lastlevel’), first(when(col(‘gender’) == “F”, 1).otherwise(0)).alias(‘gender’), first(col(‘obsstart’)).alias(‘obsstart’), first(col(‘obsend’)).alias(‘obsend’), first(col(‘endstate’)).alias(‘endstate’),  # Aggregated activity statistics count(col(‘page’)).alias(‘nact’),Fsum(when(col(‘page’) == “NextSong”, 1).otherwise(0)).alias(“nsongs”), Fsum(when(col(‘page’) == “Thumbs Up”, 1).otherwise(0)).alias(“ntbup”), Fsum(when(col(‘page’) == “Thumbs Down”, 1).otherwise(0)).alias(“ntbdown”), Fsum(when(col(‘page’) == “Add Friend”, 1).otherwise(0)).alias(“nfriend”), Fsum(when(col(‘page’) == “Add to Playlist”, 1).otherwise(0)).alias(“nplaylist”),  Fsum(when(col(‘page’) == “Submit Downgrade”, 1).otherwise(0)).alias(“ndgrade”), Fsum(when(col(‘page’) == “Submit Upgrade”, 1).otherwise(0)).alias(“nugrade”), Fsum(when(col(‘page’) == “Home”, 1).otherwise(0)).alias(“nhome”), Fsum(when(col(‘page’) == “Roll Advert”, 1).otherwise(0)).alias(“nadvert”), Fsum(when(col(‘page’) == “Help”, 1).otherwise(0)).alias(“nhelp”), Fsum(when(col(‘page’) == “Settings”, 1).otherwise(0)).alias(“nsettings”), Fsum(when(col(‘page’) == “Error”, 1).otherwise(0)).alias(“nerror”),  # Aggregated activity statistics in different periods  Fsum(when(col(‘timebeforeend’) < trend_est, 1).otherwise(0)).alias(“nact_recent”), Fsum(when(col(‘timefromstart’) < trend_est, 1).otherwise(0)).alias(“nact_oldest”), Fsum(when((col(‘page’) == “NextSong”) & (col(‘timebeforeend’) < trend_est), 1).otherwise(0)).alias(“nsongs_recent”), Fsum(when((col(‘page’) == “NextSong”) & (col(‘timefromstart’) < trend_est), 1).otherwise(0)).alias(“nsongs_oldest”) )
After completing the feature engineering step we have analysed the correlations between the built features.
# For visualization purposes we switch to pandas dataframesdf_user_pd = df_user.toPandas()# Calculate correlations between numerical featurescormat = df_user_pd[['nact_perh','nsongs_perh', 'nhome_perh', 'ntbup_perh','ntbdown_perh', 'nfriend_perh','nplaylist_perh', 'nadvert_perh', 'nerror_perh', 'upgradedowngrade', 'songratio', 'positiveratio','negativeratio', 'updownratio', 'trend_act', 'trend_songs', 'avgsessionitems',  'avgsessionlength','avgsongs']].corr()# Plot correlationsplt.rcParams['figure.figsize'] = (10,10)plt.subplots_adjust(left=0.20, right=0.9, top=0.95, bottom=0.15)sns.heatmap(cormat, cmap = "YlGnBu", square = True, vmin = -1, vmax = 1);plt.title('Feature correlations');plt.savefig('correlations.png')
The heat-map above describes a high correlation between variables nact_perh and nsongs_perh. This is expected, as listening to songs is obviously by far the most common user activity. For the same reason there is a high correlation between trend_act and trend_songs. In both cases we decided to simply remove from all further analyses and keep only variables that measure the most important interaction — playing songs.
In order to further reduce the multi-colinearity in the data, we also decided not to use nhome_perh and nplaylist_perh in the model. Moreover, avgsessionlength is highly correlated with average items in each session, hence it can be ignored too.
From the visualisations presented below the following observations have been made:
on average churned users played more songs per hour;
churned users gave significantly more thumbs down per hour, and had to watch more advertisements on average;
the ratio of songs and positive interactions relative to total activity was typically lower for churned users
churned users had on average less interactions per session
churn rate is higher for users in the free subscription plan
churn rate is slightly higher for male users
No features have been removed based on this analysis.
We first performed grid search with cross validation to test the performance of several parameter combinations, all on the user-level data obtained from the smaller Sparkify user activity data-set. Based on the performance results obtained in cross validation (measured by AUC and F1 score), we identified the best-performing model instances and retrained them on the entire training set.
maxIter (maximum number of iterations, default=100) : [10, 30]
regParam (regularization parameter, default=0.0) : [0.0, 0.1]
elasticNetParam (mixing parameter — 0 for L2 penalty, 1 for L1 penalty, default=0.0) : [0.0, 0.5]
maxDepth (maximum tree depth, default=5) : [4, 5, 6, 7]
numTrees (number of trees, default=20) : [20, 40]
maxDepth (maximum tree depth, default=5) : [4, 5]
maxIter (maximum number of iterations, default=20) : [20, 100]
In the defined grid search objects the performance of each parameter combination is by default measured by average AUC score (area under the ROC) obtained in 4-fold cross validation. AUC is briefly explained in Section 4.4 below.
numeric_columns = [‘nsongs_perh’, ‘ntbup_perh’,’ntbdown_perh’, ‘nfriend_perh’, ‘nadvert_perh’, ‘nerror_perh’, ‘upgradedowngrade’, ‘songratio’, ‘positiveratio’,’negativeratio’, ‘updownratio’, ‘trend_songs’, ‘avgsessionitems’,’avgsongs’]# Combining multiple numerical features using VectorAssemblernumeric_assembler = VectorAssembler(inputCols = numeric_columns, outputCol = “numericvectorized”)# Standardizing numerical featuresscaler = StandardScaler(inputCol = “numericvectorized”, outputCol = “numericscaled”, withStd = True, withMean = True)# Adding the two binary featuresbinary_columns = [‘lastlevel’, ‘gender’]total_assembler = VectorAssembler(inputCols = binary_columns + [“numericscaled”], outputCol = “features”)# Defining three different pipelines with three different classifiers, all with default parameters# Logistic regression lr = LogisticRegression()pipeline_lr = Pipeline(stages = [numeric_assembler, scaler, total_assembler, lr])# Random forest classifierrf = RandomForestClassifier()pipeline_rf = Pipeline(stages = [numeric_assembler, scaler, total_assembler, rf])# Gradient-boosted tree classifiergb = GBTClassifier()pipeline_gb = Pipeline(stages = [numeric_assembler, scaler, total_assembler, gb])
F1 score is preferred performance metrics to this problem. The input user-level data-set is imbalanced. Music streaming service aims to identify most of the users that are likely to churn (aiming for high recall), but at the same time does not want to give too many discounts for no reason (aiming for high precision), i.e. to users who are actually happy with the service (false positives) — this can help music streaming business prevent financial losses.
class F1score(Evaluator):def __init__(self, predictionCol = “prediction”, labelCol=”label”): self.predictionCol = predictionCol self.labelCol = labelColdef _evaluate(self, dataset):  # Calculate F1 score  tp = dataset.where((dataset.label == 1) & (dataset.prediction == 1)).count() fp = dataset.where((dataset.label == 0) & (dataset.prediction == 1)).count() tn = dataset.where((dataset.label == 0) & (dataset.prediction == 0)).count() fn = dataset.where((dataset.label == 1) & (dataset.prediction == 0)).count()  # Add epsilon to prevent division by zero precision = tp / (tp + fp + 0.00001) recall = tp / (tp + fn + 0.00001)  f1 = 2 * precision * recall / (precision + recall + 0.00001)  return f1def isLargerBetter(self): return True
The best performing model has AUC score of 0.981 and F1 score of 0.855.
As seen in the visualisation above, the most important feature for identifying churned users is the nerror_perh which measures how many error pages have been shown to the user per hour. The more errors a user has to experience the more likely it is that he/she is dissatisfied with the service.
Similar holds also for the second and third most important feature, ntbdown_perh and nadvert_perh which measure the number of thumbs down given per hour and number of advertisements seen per hour, respectively.
Most interesting feature is the trend_songs variable which measures a user’s song listening activity trend, as the fourth most important.
The Gradient-boosted tree classifier has a F1 score (precision and recall) of 0.855 and can identify churned users based on past user activity and interaction with the music streaming service, which can help the business prevent from severe financial losses.
Some improvements are to perform comprehensive grid search for the model on the full Sparkify data-set. Utilise song-level features that have been ignored so far, e.g. calculate the user’s listening diversity in terms of different songs/artists listened to in the specified observation period, etc. Build new features such as e.g. average length of song listening sessions, ratios of skipped or partially listened songs, etc.
For more details about this project, see the link to my Github available here.