How to evaluate my Classification Model results | by Songhao Wu | Towards Data Science

Confusion Matrix, ROC-AUC, F1 Score, Log Loss
Different from the Regression model, when we assess classification model results, we should not only focus on prediction results but also need to consider the prediction probability and prediction ability of each class.
There are plenty of articles online about classification metrics selection and here I will just use my own words to explain my top 5 important metrics you should consider and know before you evaluate your classification model.
1. Confusion Matrix
2. Accuracy, Recall, Precision
3. F1 Score
4. Receiving Operating Characteristics(ROC),Area under the Curve(AUC)
5. Log Loss
The list above does not follow the sequence of importance. For illustration purposes, I will use a housing data example to show the classification result on predicting the housing price range.
From the name we can tell, Confusion Matrix output is a matrix and it is quite ‘confusing’? However, it is actually quite intuitive if you understand the logic. Confusion Matrix is a 2*2 table(for binary class classification)and it is the basis of many other metrics. Assume your classification only has two categories of results (1 or 0), a confusion matrix is the combination of your prediction(1 or 0) vs actual value(1 or 0).
True Positive: Case when you correctly predict the positive result
False Positive: Case when you predict the result to be positive but it is actually negative
False Negative: Case then you predict the result to be negative but it is actually positive
True Negative: Case when you correctly predict negative result
Among the four sections, the first word indicates whether your prediction is correct and the second word indicates your prediction value. It is very useful to see how your classification model performs on different categories of value. The matrix is not just limited to 2*2 and you are able to see a bigger matrix if your classification result has more than 2 categories.
#Calculate confusion matrixfrom sklearn.metrics import confusion_matrixmatrix=confusion_matrix(y_test, y_predicted,labels=['high','low'])cm=pd.DataFrame(matrix,index=['high','low'],columns=['high','low'])print(cm)
Here I have categorized housing prices into high and low and you can see an overall of classified results in different categories. Note that in sklearn package, the y axis is the true value why x value is the predicted value which is different from the normal convention. You can only see the absolute number and it is not so direct to see the percentage metrics here.
The 3 metrics are calculated directly from the confusion matrix result and are percentage metrics based on the absolute number you see from the confusion matrix.
Accuracy is purely how much percentage of cases that you have predicted correctly overall.
Accuracy = (TP+TN)/(TP+TN+FP+FN)
Accuracy is a good measure of how the overall model performs. However, it is not telling you the performance in each category and thus you may miss important information if you purely look at accuracy. It is best to use when a number of cases in different categories are similar or used together with other metrics.
Recall measures how much percentage of real positive cases are correctly identified.
Recall = TP/(TP+FN)
Recall is used when the aim is to capture a maximum number of positive cases. For example, during the recent Covid-19 situation, the government wants to track all the infected cases in the community. If you have built a model to predict whether the person is infected with Covid-19 based on symptoms, Recall will be an important metric to be measured.
Precision measures that among the cases predicted to be positive, how much percentage of them are really positive.
Precision = TP/(TP+FP)
Precision is used when you want to be very accurate in measuring positive cases. It always conflicts with Recall because the more positive cases you want to capture, the lower the standard you will put to be classified as positive cases. In the Covid-19 cases example, if you want to capture more infected cases, you may want to include all cases that have slight symptoms that could just be because of normal flu.
#Calculate Accuracyfrom sklearn.metrics import accuracy_scoreaccuracy_score(y_test,y_predicted)# Accuracy is 0.88124999999999998#Calculate Recallfrom sklearn.metrics import recall_scorerecall_score(y_test, y_predicted)#Recall is 0.84109589041095889#Calculate Precisionfrom sklearn.metrics import precision_scoreprecision_score(y_test, y_predicted)# Precision is 0.84573002754820936
We can see that Accuracy is slightly higher than both recall and precision here because the model is predicting better on negative(low price) cases.
Considering the conflicting feature between precision and recall, the F1 Score is created to have a balanced metric between recall and precision. It is the Harmonic mean of recall and precision.
You may want to ask: why we do not simply average Precision and Recall? Well, if the distribution of positive and negative cases is very uneven, the average may not be a good representation of model performance.
In the example above, there is only 1 real positive case and the model captures it. However, among the 100 cases identified to be positive, only 1 of them is really positive. Thus, recall=1 and precision=0.01. The average between the two is 0.505 which is clearly not a good representation of how bad the model is. F1 score= 2*(1*0.01)/(1+0.01)=0.0198 and this gives a better picture of how the model performs.
#calculate F1 scorefrom sklearn.metrics import f1_scoref1_score(y_test,y_predicted)# F1 score is 0.84340659340659341
AUC is the area under the ROC curve and it is a good measure on two things:
How well the model can separate the two classes(positive and negative)How accurate is the model identify for different categories(like whether it identify group A to positive correctly)?
How well the model can separate the two classes(positive and negative)
How accurate is the model identify for different categories(like whether it identify group A to positive correctly)?
Here we need to understand two metrics first:
True Positive Rate(TPR): TPR actually equals to Recall which is among the truly positive cases, how much percentage of them are captured in the model correctlyFalse Positive Rate(NPR): This measures among the truly negative cases, how much percentage of them are actually false positive.
True Positive Rate(TPR): TPR actually equals to Recall which is among the truly positive cases, how much percentage of them are captured in the model correctly
False Positive Rate(NPR): This measures among the truly negative cases, how much percentage of them are actually false positive.
TPR and NPR are positively correlated. This is because when you want to capture more positive cases in your classification model, inevitably you will misclassify some of them, and NPR increases. ROC curve is TPR plotted against NPR and the bigger the area under the curve the better the separation between different groups.
#Calculate AUC from sklearn.metrics import roc_curve, aucfpr, tpr, treshold = roc_curve(y_test, y_score_new)roc_auc = auc(fpr, tpr)#auc is 0.96
For classification problems on 2 classes (either 1 or 0), the probability of you blindly guessing is 50%. The blue dotted line shows the curve TPR and FPR when you blindly guess the category and for that diagonal line, the area under the curve(AUC) is 0.5.
Each point on the curve represents a different cut-off point between TPR and FPR. For example, if I cannot tolerate any false positive rate (<0.01), then the true positive rate I can reach is around 0.6(see the green dot on the graph). If I loosen my criteria a bit and I only need to control FPR below 0.1, then my TPR can reach 0.9(see the red dot on the graph).
The selection of threshold is based on different business requirements and usually, the best choice is to maximize the difference between TPR and FPR, and on the graph is represented by the maximum vertical distance between the orange and blue dotted line(see the red line). Our prediction example has a pretty good AUC(0.96) and the optimized point is around (0.1,0.9)
You can find more information on ROC-AUC in the link below:
towardsdatascience.com
Log loss function penalizes wrongly classified cases and also lacks of confidence in correctly-classified cases. Same as ROC-AUC, it does not only consider the classification accuracy or results but also considers the probability assigned to each of the cases from the model. The better the model, the lower the log loss value.
y is the real result(0 or 1) and p function is the probability that the model predicts the outcome to 1. Thus, if the real value is 1, we want to maximize the probability that assigns to the prediction of 1 which is p(y) while if the real value is 0 we want to maximize the probability of predicting 0 which is 1-p(y). That is the part after the summation sign which we want to maximize. log function of value between 0–1 is negative. Thus, to make the whole value positive the function has a minus sign at the front and we need to minimize the whole function.
Log Loss value can vary from 0 to infinity. However, if you calculate according to the formula, the log loss function is 0.69 if you just blindly assign 50% to each case. Thus, you should definitely keep the log loss value below 0.69 and decide the threshold based on business requirements.
#Calculate log lossfrom sklearn.metrics import log_losslog_loss(y_test,y_score)#log loss is 0.32241485496779987
In conclusion, different metrics serve different purposes and can be applied accordingly to different problems. However, it is advised to use ROC-AOC or log loss generally because they consider the prediction probability of the prediction for each class. Even though both models predict correctly for the case, the 90% confidence prediction model is definitely more preferred than the 51% confidence prediction model.
If you are interested to know what metrics to use for the regression model, you can refer to the link below: