Finding Needle in the Haystack with Apache Spark
By: Mehmet Öner Yalçın
Tags: Churn, Predictive Analytics, Big Data, Apache Spark, Data Science

According to the two tables below, firstName , gender , lastName , location , registration , userAgent and auth cols have NULL values when users are logged out or listening as Guest. Since we are really interested in users who are registered and likely to churn, we can remove these rows as they are not logged in users.

data.filter(data.firstName.isNull())\

.groupBy('firstName','gender', 'lastName',

'location', 'registration','userAgent', 'auth')\

.count()\.show()

data.groupBy('auth').count().show()

Rows where firstName .. etc are null they all are either Guest or Logged out.

# Drop the rows with null values for the columns below

data = data.na.drop(subset=['firstName','gender',

'lastName','location','registration',

'userAgent', 'auth']) sns.heatmap(data.toPandas().isnull())

Removing logged out users gives us a cleaner dataset

Now we have a cleaner dataset where users are logged in.

ts column is the timestamp in unix epoch format, we better convert into timestamp format, and add it into timestamp column

data = data\

.withColumn('timestamp', from_unixtime(data.ts/1000)) data.take(2) [Row(artist='Martha Tilston', auth='Logged In', firstName='Colin', gender='M', itemInSession=50, lastName='Freeman', length=277.89016, level='paid', location='Bakersfield, CA', method='PUT', page='NextSong', registration=1538173362000, sessionId=29, song='Rockpools', status=200, ts=1538352117000, userAgent='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0', userId='30', timestamp='2018-10-01 00:01:57'), Row(artist='Five Iron Frenzy', auth='Logged In', firstName='Micah', gender='M', itemInSession=79, lastName='Long', length=236.09424, level='free', location='Boston-Cambridge-Newton, MA-NH', method='PUT', page='NextSong', registration=1538331630000, sessionId=8, song='Canada', status=200, ts=1538352180000, userAgent='"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36"', userId='9', timestamp='2018-10-01 00:03:00')]

For our model, we will be dropping some fields as we do not need them, maybe they have duplicate information in other column or just not relevant. At this point I decided not to use them in my model, but since model building is an iterative process I may come later and decide not to drop any of these columns if I believe adding these will make a difference.

cols_to_drop = ['userAgent', 'artist', 'firstName',

'lastName', 'location', 'song',

'ts', 'registration', 'length',

'method', 'status'] data = data.drop(*cols_to_drop) data.show()

User music after adding timestamp and removing irrelevant columns for prediction

The table below tells us, both free and paid users can cancel their services, but only paid customers can downgrade services. Well, that makes sense. I believe we can further filter our dataframe to clean few rows as pages like Settings or Help won’t tell much about churn.

Pages vs User Level. Next Song is the dominant as expected. There are 52 Cancellations (churns)

pages_to_keep = ['NextSong', 'Downgrade',

'Cancellation Confirmation', 'Upgrade',

'Add to Playlist', 'Cancel', 'Submit Upgrade', ] data = data.filter(col('page').isin(pages_to_keep)) data.groupBy('auth', 'page').count().orderBy('auth').show()

After filtering out pages like Help, we are left with these above. Cancellation Confirmation is the churn KPI

Exploratory Data Analysis

Cancellation Confirmation is the most clear KPI for churn. Downgrade page is accessed hundreds of times but, data doesn't clearly tell us if users downgraded after accessing this page, such as we do not have Downgrade Confirmed followed by a Downgrade page. Also the number of Downgrade row in the dataset is far bigger than the unique users. Therefore I sticked to the Cancellation Confirmation as the churn indicator.

# In our final dataset we do not have any null values

data.select([count(when(col(c).isNull(), c)).alias(c)

for c in data.columns])\

.head()\

.asDict() {'auth': 0,

'gender': 0,

'itemInSession': 0,

'level': 0,

'page': 0,

'sessionId': 0,

'userId': 0,

'timestamp': 0}

How many distinct users do we have in our dataset?

data.select('userId').distinct().count() >> 225

Let’s have a look at a random user’s listening activity. itemInSession increments monotonically for a continuous listening session. We may extract the max value for each session to understand user engagement (later in Feature Engineering section)

data.filter(col('userId') == 30).show(20)

User activity for user with userId 30. Note that itemInSession increases when the page changes. sessionID stays the same for a continuous music streaming session.

Let’s have brief look at churned users/susbcribers. Looks like both free and paid customers can churn. We save this data to churn_data as we will use this in later when joining in Feature Engineering section

churn_data = data.filter(col('page')=='Cancellation Confirmation')

churn_data.show(10)

List of churned users, there are both free and paid customers. We store the userId for joining later with other summarised dataset

We’ll later need a list of churned users when joining datasets in Feature engineering section, create a new column names churned

churn_data_summary = churn_data.select('userId')\

.withColumn('churned', F.lit(True)) churn_data_summary.show(10)

Churned userIds

Feature Engineering

Here in this section we will be building few features/predictors in order to predict churned users. This in general means creating new columns using existing columns and transforming so that we can feed into ML pipeline.

Mean time between sessions

One useful feature to measure user engagement would be to see how frequently users are listening Sparkify platform. We look at how many sessions a user has as well as mean time between sessions. Users are likely to be engaged more if they are visiting our app/platform frequently, therefore mean time between sessions should be shorter for engaged users.

userId 10 has 6 streaming sessions and mean time between sessions is 7 days.

In our model, we will only need the sessionsFreqDay as our feature and userId is the key for joining other datasets below, remanining columns are not necessary so we are filtering out

listen_freq = listen_freq.select('userId', 'sessionsFreqDay') listen_freq.show(10)

we use userId as key when joining with other data. sessionsFreqDay is a new feature engineered predictor

Session Count, Average Song Count/Session

Another two good predictors would be the total session count for a given user and the average song count listened per session. Users are likely to be more engaged if they have more sessions and if for each session they listened to many songs.

Below is another aggregate dataset showing each user’s listening activity summary, how many sessions they listened and average song count for all sessions.

user with id 10, has 6 sessions and on average he/she listened to 133.7 songs/session, whereas user with id 100006 has only 1 session and listened to 45 songs. User 100006 is engaged less.

Below is the SQL equivalent of the pyspark code above for user_engagement , I just added here for reference, it exactly gives the same data, but use a bit easier to read for reference. For those who prefer SQL over Dataframe approach will be happy to see they can use almost only SQL for data manipulation.

Same result is achieved with SQL

User Features

User features like gender and level could be useful when predicting churn, as behaviour of free and paid or female and male users might be different. Let's save these features into user_feature dataframe

user_features = data.select('gender', 'userid', 'level')\

.distinct().orderBy('userId') user_features.show(10)

User Features

Finally, let’s join all the features we have created above into a joined dataframe

churned column is the target variable and all other columns (except userId) are predictors

Now we have 5 predictor columns for churn:

gender : M or F

: M or F level : paid or free

: or sessionCount : How many music sessions user listened to? Each session has multiple songs

: How many music sessions user listened to? Each session has multiple songs meanSongCount : For given sessions, what is the average number of songs each user listened to? (How long a session lasts)

: For given sessions, what is the average number of songs each user listened to? (How long a session lasts) sessionFreqDay : What is the average day count between sessions? Lower means more engagement.

Before moving further, let’s do a sanity check and see if our predictors make sense. Below is the mean value for sessionCount , meanSongCount and SessionFreqDay for each churned and gender group

joined.groupBy('churned','gender')\

.agg({'sessionCount': 'mean',

'meanSongCount': 'mean',

'sessionsFreqDay': 'mean'})\

.orderBy('churned','gender')\

.show()

According to table above churned users tend to have lower sessionCount s and they listen to less music in sessions, but engages with platform more frequently (2 - 2.5 days between sessions for churned users and 4 - 4.8 days for non-churned users). The last observation is a bit contrary to our expectation, but nevertheless this is a useful predictor. One interesting finding is that female users are more engaged with the platform than male users, this will be also useful for our model

joined.groupBy('churned','level')\

.agg({'sessionCount': 'mean',

'meanSongCount': 'mean',

'sessionsFreqDay': 'mean'})\

.orderBy('churned','level')\

.show()

When we look at the data from the perspective of free or paid users, we have similar observations. sessionCount and meanSongCount both are higher for non-churned users as expected, however again churned users engaged with the platform more frequently. Does this mean they churned not because of their lack of interest but because they are annoyed they cannot find the musics they are looking for and left the platform because of that? This would be interesting idea to explore for later. But we won’t have a look at it in this blog post. It is always a good habit to ask questions and if there is a contradictory finding, ask more questions to discover underlying fact.

Another reason why we had a contradictory finding could be simply statistical quirk. Simpson’s paradox may haunt us in some cases. Just to make sure we are not falling into the statistical fallacy of Simpson’s paradox let’s have a look at both level and gender together.

joined.groupBy('churned','level', 'gender')\

.agg({'sessionCount': 'mean',

'meanSongCount': 'mean',

'sessionsFreqDay': 'mean'})\

.orderBy('churned','level', 'gender')\

.show()