Metrics and Python II. In the previous article, we take a... | by Gabriel Naya | Towards Data Science

We make a dataset with three arrays: real values, predicted values, and likelihood values.real_values: A data set with 1000 elements between 0 and 1.pred_values: A variation of the real dataset, emulates a prediction, changing only the first 150 values.prob_values: Is the pred_values array but contains the percent of probability of being zero or one, instead of the real value 0 or 1. It will be used for ROC charts and Recall versus Precision.
real_values = []prob_values = []pred_values = []for k in range(0,1000):   value = random.uniform(0, 1)   real_values.append(int(round(value,0)))   if k < 150:      value2 = random.uniform(0, 1)      prob_values.append(value2)      pred_values.append(int(round(value2,0)))   else:      prob_values.append(value)      pred_values.append(int(round(value,0)))
The Confusion Matrix is not a metric in itself but is an essential tool for classifying real data and predictions, so its components are the trigger for the first set of metrics.
from sklearn.metrics import confusion_matriximport scikitplot as skpltprint(confusion_matrix(real_values, pred_values))skplt.metrics.plot_confusion_matrix(real_values, pred_values,figsize=(8,8))
You can download complete file in https://kreilabs.com/wp-content/uploads/2019/12/Metrics_table.pdf
Here we can see pyhton implementation for table metrics in matrix_metrix routine:
import sklearn.metricsimport mathdef matrix_metrix(real_values,pred_values,beta):   CM = confusion_matrix(real_values,pred_values)   TN = CM[0][0]   FN = CM[1][0]    TP = CM[1][1]   FP = CM[0][1]   Population = TN+FN+TP+FP   Prevalence = round( (TP+FP) / Population,2)   Accuracy   = round( (TP+TN) / Population,4)   Precision  = round( TP / (TP+FP),4 )   NPV        = round( TN / (TN+FN),4 )   FDR        = round( FP / (TP+FP),4 )   FOR        = round( FN / (TN+FN),4 )    check_Pos  = Precision + FDR   check_Neg  = NPV + FOR   Recall     = round( TP / (TP+FN),4 )   FPR        = round( FP / (TN+FP),4 )   FNR        = round( FN / (TP+FN),4 )   TNR        = round( TN / (TN+FP),4 )    check_Pos2 = Recall + FNR   check_Neg2 = FPR + TNR   LRPos      = round( Recall/FPR,4 )    LRNeg      = round( FNR / TNR ,4 )   DOR        = round( LRPos/LRNeg)   F1         = round ( 2 * ((Precision*Recall)/(Precision+Recall)),4)   FBeta      = round ( (1+beta**2)*((Precision*Recall)/((beta**2 * Precision)+ Recall)) ,4)   MCC        = round ( ((TP*TN)-(FP*FN))/math.sqrt((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))  ,4)   BM         = Recall+TNR-1   MK         = Precision+NPV-1   mat_met = pd.DataFrame({'Metric':['TP','TN','FP','FN','Prevalence','Accuracy','Precision','NPV','FDR','FOR','check_Pos','check_Neg','Recall','FPR','FNR','TNR','check_Pos2','check_Neg2','LR+','LR-','DOR','F1','FBeta','MCC','BM','MK'],     'Value':[TP,TN,FP,FN,Prevalence,Accuracy,Precision,NPV,FDR,FOR,check_Pos,check_Neg,Recall,FPR,FNR,TNR,check_Pos2,check_Neg2,LRPos,LRNeg,DOR,F1,FBeta,MCC,BM,MK]})   return (mat_met)
When we call matrix_metrix function:
beta = 0.4mat_met = matrix_metrix(real_values,pred_values,beta)print (mat_met)
TP = True positive
TN = True negative
FP = False positive — Equivalent with false alarm or Type I error1
FN = False negative — Equivalent with miss or Type II error 2
Prevalence: Serves to measure the balance of data within the total population. It is possible to measure the prevalence of positives or negatives and the sum of both quotients is = 1, a balanced data set would give coefficients close to 0.5If, on the contrary, one of the factors is close to 1 and the other to 0, we are going to have an unbalanced data set.
Accuracy: It is the measure of success of the system, the total of positive and negative achievements over the total population, indicates the degree of success of the model. According to the cost of the sensitivity3 of the case study, and the balance of the data (prevalence).
PPV — Positive predictive value o precisión
NPV — Negative predictive value
The PPV and NPV describe the performance of a diagnostic test or other statistical measure. A high result can be interpreted as indicating the accuracy of such a statistic. The PPV and NPV are not intrinsic to the test; they depend also on the prevalence. 4
FDR — False Discovery rate (Type I) [iv]
FOR — False Omission Rate
check_Pos: PPV + FDR = 1
check_Neg: FOR + NPV = 1
Recall, sensitivity, probability of detection, and power are the same measure, which assumes different names and applications depending on the field of study.In the literature referring to binary classification problems, the use of the term recall is more frequent.
“Sensitivity and specificity are statistical measures of the performance of a binary classification test, also known in statistics as a classification function, that are widely used in medicine:
· Sensitivity measures the proportion of actual positives that are correctly identified as such.
· Specificity (also called the true negative rate) measures the proportion of actual negatives that are correctly identified as such (e.g., the percentage of healthy people who are correctly identified as not having the condition).5
FPR — false positive ratio (or false alarm ratio) is the probability of falsely rejecting the null hypothesis for a particular test.
FNR — false negative rate is the proportion of positives which yield negative test outcomes with the test, i.e., the conditional probability of a negative test result given that the condition being looked for is present.
In statistical hypothesis testing, this fraction is given the letter β. The “power” (or the “sensitivity”) of the test is equal to 1−β.
Not to be confused with Likelihood-ratio test.
LR (+ -) Likelihood ratios: They are used for assessing the value of performing a diagnostic test. 6
DOR — diagnostic odds ratio is a measure of the effectiveness of a diagnostic test 7
From the matrix_metrix function used for Table 1, we also got the values for this Table 2:
In statistical analysis of binary classification, the F1 score (also F-score or F-measure) is a measure of a test’s accuracy. It considers both the precision p and the recall r of the test to compute the score: p is the number of correct positive results divided by the number of all positive results returned by the classifier, and r is the number of correct positive results divided by the number of all relevant samples (all samples that should have been identified as positive). The F1 score is the harmonic mean of the precision and recall, where an F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.8
Beta is a parameter for score analysis, if we use Beta equal to 1, we have the harmonic mean; then we have the F1 score:
We could take some Rules of Thumb:
To give more weight to the Precision, we must select a Beta value between 0–1 0 < Beta < 1
To give more weight to the Recall, we pick a Beta Value in the interval 1 < Beta < +infinite
We can find an excellent and very complete article regarding the concepts and implementation of these instruments.
ROC — Receive operating characteristic
It is a plot of the false positive rate (x-axis) versus the true positive rate (y-axis) for a number of different candidate threshold values between 0.0 and 1.0. Put another way, it plots the false alarm rate versus the hit rate.
Precision-recall curves
A precision-recall curve is a plot of the precision (y-axis) and the recall (x-axis) for different thresholds, much like the ROC curve. A no-skill classifier is one that cannot discriminate between the classes and would predict a random class or a constant class in all cases. The no-skill line changes based on the distribution of the positive to negative classes. It is a horizontal line with the value of the ratio of positive cases in the dataset. For a balanced dataset, this is 0.5.
AUC — Area under curve
The area under the curve (AUC) can be used as a summary of the model skill.
We based on the mentioned article for the following code
#ROC Implementationfrom sklearn.metrics import roc_curvefrom sklearn.metrics import roc_auc_scorefrom matplotlib import pyplotfpr, tpr, thresholds = roc_curve(real_values, prob_values)auc = roc_auc_score(real_values, prob_values)print('AUC: %.3f' % auc)pyplot.plot(fpr, tpr, linestyle='--', label='Roc curve')pyplot.xlabel('False Positive Rate')pyplot.ylabel('True Positive Rate')pyplot.legend()pyplot.show()
#Precision-recall implementationprecision, recall, thresholds = sklearn.metrics.precision_recall_curve(real_values,prob_values)pyplot.plot(recall, precision, linestyle='--', label='Precision versus Recall')pyplot.xlabel('Recall')pyplot.ylabel('Precision')pyplot.legend()pyplot.show()
Here a simple and fast report about binary classification values
And here a function for get many metrics directly from sklearn:
def sk_metrix(real_values,pred_values,beta):   Accuracy = round( sklearn.metrics.accuracy_score(real_values,pred_values) ,4)      Precision  = round( sklearn.metrics.precision_score(real_values,pred_values),4 )      Recall     = round( sklearn.metrics.recall_score(real_values,pred_values),4 )   F1         = round ( sklearn.metrics.f1_score(real_values,pred_values),4)      FBeta      = round ( sklearn.metrics.fbeta_score(real_values,pred_values,beta) ,4)   MCC        = round ( sklearn.metrics.matthews_corrcoef(real_values,pred_values)  ,4)   Hamming    = round ( sklearn.metrics.hamming_loss(real_values,pred_values) ,4)   Jaccard    = round ( sklearn.metrics.jaccard_score(real_values,pred_values) ,4)   Prec_Avg   = round ( sklearn.metrics.average_precision_score(real_values,pred_values) ,4)   Accu_Avg   = round ( sklearn.metrics.balanced_accuracy_score(real_values,pred_values) ,4)   mat_met = pd.DataFrame({'Metric': ['Accuracy','Precision','Recall','F1','FBeta','MCC','Hamming','Jaccard','Precision_Avg','Accuracy_Avg'],'Value': [Accuracy,Precision,Recall,F1,FBeta,MCC,Hamming,Jaccard,Prec_Avg,Accu_Avg]})   return (mat_met)
Calling sk_metrix function
sk_met = sk_metrix(real_values,pred_values,beta)
The number of metrics arising from the confusion matrix is quite large and may be challenging to cover to refine the use of each indicator according to the use case. The most common metrics may be F1-Score, ROC, precision-recall AUC, prevalence, and sensitivity.
This series of articles intended to create a table in which to quickly have a guide of the majority of metrics used for each case study; mastering their application exceeds our scope and also seems to be quite difficult.Remember that the table can be download in pdf format.
In the next article, we will see more classification metrics and specific metrics for recommendation systems in table 2.3
[*] https://en.wikipedia.org/wiki/Confusion_matrix
[1] type i error
[2] type ii error
[3] https://medium.com/mercadolibre-datablog/cost-sensitive-classification-in-fraud-prevention-263170d8fcfe
[4] The positive and negative predictive values (PPV and NPV respectively) are the proportions of positive and negative results in statistics and diagnostic tests that are true positive and true negative results, respectively.[1] “The PPV and NPV describe the performance of a diagnostic test or other statistical measure. A high result can be interpreted as indicating the accuracy of such a statistic. The PPV and NPV are not intrinsic to the test; they depend also on the prevalence.[2] The PPV can be derived using Bayes’ theorem. Although sometimes used synonymously, a positive predictive value generally refers to what is established by control groups, while a post-test probability refers to a probability for an individual. Still, if the individual’s pre-test probability of the target condition is the same as the prevalence in the control group used to establish the positive predictive value, the two are numerically equal.