Evaluating Performance of Models. Log reg/classification evaluation... | by Michelle Venables | Towards Data Science

After completing some data science projects in logistic regression and binary classification I have decided to write more about the evaluation of our models and steps to take to make sure they are running efficiently and accurately. You may have “good” data and understand how to build a model, but if you are able to interpret the efficiency of your model you’ll be able to interpret your results and provide the ultimate business decisions!
With regression, we predict continuous variables so it makes sense to discuss error as a distance of how far off our estimates are. With classification problems, we are dealing with binary variables so our model is either right or wrong. As a result, we examine a few specific measurements when evaluating performance of a classification algorithm.
Precision — measures how precise the predictions are.Recall — indicates what percentage of the classes we’re interested in were actually captured by the model.Accuracy is probably the most intuitive metric. We use it to measure the total number of predictions a model gets right, including both True Positives and True Negatives.F1 score is a measurement that considers both precision and recall to compute the score. It can be interpreted as a weighted average of the precision and recall values and cannot be high without both precision and recall also being high. When a model’s F1 score is high, you know that your model is doing well all around.
Precision — measures how precise the predictions are.
Recall — indicates what percentage of the classes we’re interested in were actually captured by the model.
Accuracy is probably the most intuitive metric. We use it to measure the total number of predictions a model gets right, including both True Positives and True Negatives.
F1 score is a measurement that considers both precision and recall to compute the score. It can be interpreted as a weighted average of the precision and recall values and cannot be high without both precision and recall also being high. When a model’s F1 score is high, you know that your model is doing well all around.
Example: Predict if an employee is Employed or Not Employed (Active). In this project we ideally what to predict if a candidate is a good fit for the company.
Precision: “Out of all the times the model said someone was employed, how many times did the employee in question actually have a job at “Dental Magic?”
Recall: “Out of all the employees we saw that actually were employed, what percentage of them did our model correctly identify as having active employment?”
Precision and recall have an inverse relationship. As one goes up the other will go down.
This is a classic data science question. What is better more false positives or false negatives? The answer is “IT DEPENDS ON THE PROBLEM” and can test your critical thinking abilities. So let’s examine a new example.
Detecting credit card fraud, a false positive would be when our model flags a transaction as fraudulent and it isn’t. This results in a slightly annoyed customer. A false negative might be a fraudulent transaction that a company mistakenly lets through as normal consumer behavior. In this case, the credit card company could be on the hook for reimbursing the customer for thousands of dollars because they missed the signs that the transaction was fraudulent! Although being wrong is never ideal, it makes sense that credit card companies tend to build their models to be a bit too sensitive, because having a high recall saves them more money than having a high precision score.
Let’s analyze our HR example. Our model wants to predict whether a person will fit within the company “employed vs non-employed”. In this situation we ideally want both to be low: false negatives and false positives. As an HR employee we’d like our false positives to be higher than false negatives and here’s why. With a higher false positive our model predicts our candidate is a good fit when they are actually not... This is just the first round of interviewing scans of candidates, we would rather look through more resumes than miss out on one potentially good candidate for the role. The ultimate hiring decision comes down a persons final decision.
When evaluating models we want to visualize an ROC Curve. Illustrating the true positive rate against the false positive rate of our classifier. True positive rate is another name for recall which is a ratio of the true positive predictions compared to all values that are actually positive.
TPR = TP/(TP+FN)
False positive rates is the ratio of false positive predictions compared to all values that are actually negative.
FPR = FP/(FP+TN)
If TPR is our y-axis and FPR is our x-axis, we want our ROC curve to hug the left side of our chart meaning higher the TPR the lower the FPR.
The logistic regression model produces probabilities that each observation’s specific class. Imagine it looking like the image below:
If we alter cutoff point it will sacrifice precision, increasing the FPR in order to increase the TPR or vice versa. Shifting the decision boundary to the left from 0.5 will result in capturing more of the positive class (1) cases. It will also pick up some false positives (red cases at the far right of the negative (0) case distribution).
Models with poor ROC might have large overlaps in the probability estimates for the two classes. This will indicate that our model had difficulty separating the two classes from each other.
Let’s see an example:
#Import roc_curve, aucfrom sklearn.metrics import roc_curve, auc#Calculate the probability scores of each point in the training sety_train_score = model_log.decision_function(X_train)# Calculate the fpr, tpr, and thresholds for the training settrain_fpr, train_tpr, thresholds = roc_curve(y_train, y_train_score)#Calculate the probability scores of each point in the test sety_test_score = model_log.decision_function(X_test)#Calculate the fpr, tpr, and thresholds for the test settest_fpr, test_tpr, test_thresholds = roc_curve(y_test, y_test_score)
Using Matplotlib we want to visualize our ROC Curve:
plt.plot(test_fpr, test_tpr, color='darkorange', lw=lw, label='ROC curve')
As shown to the left you can see that our classification models TPR vs FPR. Think about the scenario of this model: predicting employees fit within the company. If you tune the current model to have an 76% True Positive Rate, (you’ve still missed 29% of those who would be a good fit), what is the False positive rate?
When TPR = 75% then the FPR is 22%. The true positive rate determines the percentage of employees that are active who are who are correctly identified as a good fit. At the same time, this has a FPR of nearly .25 meaning that roughly one in four times we incorrectly warn an employee of being a good fit when they are actually not performing well.
Test AUC: 0.8041
The overall accuracy of a classifier can be quantified by the AUC, the Area Under the Curve. Perfect classifiers would have an AUC score of 1.0 while an AUC of 0.5 is worthless. Here we have and AUC of .80 which seems to be accurate.
An Introduction to Statistical Learning. Gareth James Trevor Hastie Robert Tibshirani Daniela Witten
Michelle Venables — Github