Predict Employee Churn with Machine Learning
By: Col Jung
Tags: Machine Learning, Classification, Data Science, Employee Retention, Churn Prediction

4. Evaluation

4.1 Performance scores

We’ll start by printing the cross-validation scores. This is the average performance across the 10 hold-out folds and is a way to get a reliable estimate of the model performance using only your training data.

for name, model in fitted_models.items():

print(name, model.best_score_) Output:

l1 0.9088324151412831

l2 0.9088324151412831

rf 0.9793851075173272

gb 0.975475386529234

Moving onto the test data, we’ll:

calculate accuracy ;

; print the confusion matrix and calculate precision, recall and F1-score ;

and calculate and ; display the ROC and calculate the AUROC score.

Accuracy measures the proportion of correctly labelled predictions, however it is an inappropriate metric for imbalanced datasets, e.g. email spam filtration (spam vs. not spam) and medical testing (sick vs. not sick). For instance, if our dataset only had 1% of employees satisfying target=Left, then a model that always predicts the employee is still working at the company would instantly score 99% accuracy. In these situations, precision or recall is more appropriate. Whichever you use often depends on whether you want to minimise Type 1 errors (False Positives) or Type 2 errors (False Negatives). For spam emails, Type 1 errors are worse (some spam is OK as long as you don’t accidentally filter out an important email!) while Type 2 errors are unacceptable for medical testing (telling someone they didn’t have cancer when they did is a disaster!). The F1-score gets you the best of both worlds by taking the weighted average of precision and recall.

The area under the ROC, known as the AUROC is another standard metric for classification problems. It’s an effective measurement of a classifier’s ability to distinguish between classes and separate signal from noise. This metric is also robust against imbalanced datasets.

Here is the code to generate these scores and plots:

for name, model in fitted_models.items():

print('Results for:', name)



# obtain predictions

pred = fitted_models[name].predict(X_test) # confusion matrix

cm = confusion_matrix(y_test, pred)

print(cm) # accuracy score

print('Accuracy:', accuracy_score(y_test, pred))



# precision

precision = cm[1][1]/(cm[0][1]+cm[1][1])

print('Precision:', precision)



# recall

recall = cm[1][1]/(cm[1][0]+cm[1][1])

print('Recall:', recall)



# F1_score

print('F1:', f1_score(y_test, pred))



# obtain prediction probabilities

pred = fitted_models[name].predict_proba(X_test)

pred = [p[1] for p in pred] # plot ROC

fpr, tpr, thresholds = roc_curve(y_test, pred)

plt.title('Receiver Operating Characteristic (ROC)')

plt.plot(fpr, tpr, label=name)

plt.legend(loc='lower right')

plt.plot([0,1],[0,1],'k--')

plt.xlim([-0.1,1.1])

plt.ylim([-0.1,1.1])

plt.ylabel('True Positive Rate (TPR) i.e. Recall')

plt.xlabel('False Positive Rate (FPR)')

plt.show()



# AUROC score

print('AUROC:', roc_auc_score(y_test, pred))

Logistic regression (L1-regularised):

Output:

[[2015 126]

[ 111 562]]



Accuracy: 0.9157782515991472

Precision: 0.8168604651162791

Recall: 0.8350668647845468

F1: 0.8258633357825129

AUROC: 0.9423905869485105

Logistic regression (L2-regularised):

Output:

[[2014 127]

[ 110 563]]



Accuracy: 0.9157782515991472

Precision: 0.8159420289855073

Recall: 0.836552748885587

F1: 0.8261188554658841

AUROC: 0.9423246556128734

Gradient-boosted tree:

Output:

[[2120 21]

[ 48 625]]



Accuracy: 0.9754797441364605

Precision: 0.9674922600619195

Recall: 0.9286775631500743

F1: 0.9476876421531464

AUROC: 0.9883547910913578

Random forest:

Output:

[[2129 12]

[ 45 628]] Accuracy: 0.9797441364605544

Precision: 0.98125

Recall: 0.9331352154531947

F1: 0.9565879664889566

AUROC: 0.9916117990718256

The winning algorithm is the random forest with an AUROC of 99% and a F1-score of 96%. This algorithm has a 99% chance of distinguishing between a Left and Employed worker… pretty good!

Out of 2814 employees in the test set, the algorithm:

correctly classified 628 Left workers (True Positives) while getting 12 wrong (Type I errors), and

correctly classified 2129 Employed workers (True Negatives) while getting 45 wrong (Type II errors).

FYI, here are the hyperparameters of the winning random forest, tuned using GridSearchCV.