Do Paying Users Stay Longer?
By: Eszter Rékasi
Tags: Pyspark, Spark, Music Streaming App, Predictive Modeling, Churn Prediction

2.5. Evaluate results — to determine the best model

F1 score is the harmonic mean of precision and recall therefore it enables us to optimize for the precision and the robustness of a model at once. While precision tells us the proportion of users being correctly predicted as churners recall tells us the proportion of true churners successfully found.

Confusion matrix for the churn label prediction — image by author

Although F1 score was the preferable metric for so an imbalanced dataset (remember that only 23% of the users were churners) I calculated all the below evaluators for the models:

Recall (or sensitivity) = TP / (TP + FN)

Precision = TP / (TP + FP)

F1 score = 2 * Precision * Recall / (Precision + Recall)

Accuracy = (TP + TN) / All

From a business perspective, it is recall that is critical in churn prediction cases as the business might want to find as many of the potential churners as possible — even if some false positive findings might end up in the pool of predicted churners.

Model evaluation results turned out to be different for different runs of the code as I did not fix the seed either for the train-test split or the random forest models.

Evaluation of model performance (first running)

At a first running of the script, the Logistic Regression and the first Random Forest models performed equally good although none of these performed objectively very well (62% for recall and 73% for F1). The exactly same performance might be explained by the very narrow test dataset (0.2 x 225 = 4.5 records) while the poor model performance by the sample size and the simple models applied.

Evaluation of model performance (second running)

At the second running of the above script, the model performance results were somewhat different (as to be seen above in the printed results). Given that it is crucial for a streaming business to identify probable churners and act against their future churn I recommend choosing model based on its recall metrics. As a comprehensive metrics, using F1 score is more reasonable than using accuracy as the label is not balanced in our case (only 23% churners in the dataset). Based on this, Logistic Regression seems to be the best (least worse in this case) model to predict customer churn.

Evaluation of model performance (third running)

At a third run, when I also added hypertuned models I again received different results for the primary three models. As the above table summarizes, hypertuning the parameters of the model did not improve the performance of the Logistic Regression models with respect to the primary metric, F1 score (80% compared to the previously achieved 84% in case of Logistic Regression). These differences in metrics might be within the maximum tolerance for statistical error given the randomity in splitting the data, training the models and the modest sample size.

Evaluating which features played a key role in predicting the churn label according to the above models:

Chart 6: Feature importance based on the Random Forest model — top features (first run)

Some of the formally most important features do not help better understand customer behavior such as:

timestamp_registration

timestamp_last .

While a more detailed analysis - only on a significantly larger sample - could explore patterns in the behavior of earlier vs. later adopters of the platform, in our case, the associated feature importance is probably just a bias due to the extent of available data (certainly, non-churners seem to stay longer in a given timeframe). This could be balanced if the churner and non-churner user base would be matching regarding the date of registration.

The top important features that seem reasonable are the following:

visit_frequency (how often a user uses Sparkify) — Who uses Sparkify rarely is probably more prone to churn.

(how often a user uses Sparkify) — Who uses Sparkify rarely is probably more prone to churn. event_length_min and session_length_max (what the minimum length of an event is for the user) — While we can assume that users who are listening to music for a very long time are more satisfied with the music streaming experience therefore the maximum length of sessions might be an indicator for their willingness to stay with Sparkify. The minimum length of an event, on the other hand, is less self-evident. E.g. it could mean that the user was dissatisfied that is why she jumped to another screen but it also could mean that the user can easily navigate through the application.

and (what the minimum length of an event is for the user) — While we can assume that users who are listening to music for a very long time are more satisfied with the music streaming experience therefore the maximum length of sessions might be an indicator for their willingness to stay with Sparkify. The minimum length of an event, on the other hand, is less self-evident. E.g. it could mean that the user was dissatisfied that is why she jumped to another screen but it also could mean that the user can easily navigate through the application. Save Settings (proportion of going to the Save Settings page among the sum of page visits) — Probably, who invested time to explore the taylorization of the app liked it more and was less prone to churn.

(proportion of going to the Save Settings page among the sum of page visits) — Probably, who invested time to explore the taylorization of the app liked it more and was less prone to churn. max_gap_days (maximum number of days between two subsequent visits of Sparkify) and session_count (how many times a user used Sparkify) — These attributes reveal how often a user uses Sparkify. Maybe, who uses it less frequently is tend to churn more probably.

To better understand the probable direction of the impact a feature might have on the label outcome one can take a hint from the evaluation of the feature coefficients from the Logistic Regression model:

Chart 7: Top positive feature coefficients (from the Logistic Regression)

I case of features with a positive coefficient, the larger is the value of that feature for a user the more probable is that the user is a churner.

Chart 7: Top negative feature coefficients (from the Logistic Regression)

In case of features with a negative coefficient, the larger the absolute value of a feature is the less probable is that the user is a churner.