A framework for analyzing skew in evaluation metrics

For several evaluation metrics for classification problems, correctly classifying an additional point from one class will have a different effect on the value of the evaluation metric compared to correctly classifying an additional point from another class. In this paper, we describe a method for quantifying these effects based on “ metric skew”. After describing how to find the skew for each class given a particular evaluation metric, we show what the skews are for several common evaluation metrics. In particular, we show that these skews provide a new viewpoint on metrics from which previously known as well as new properties about several popular metrics can be observed. Introduction Many evaluation metrics used to analyze the results in classification problems do not weight classifications from all classes equally. The most obvious example of this are evaluation metrics used in cost-sensitive learning which specifically take disparate misclassification costs into account. However, several metrics that do not use misclassification costs are also not symmetric with respect to the effect of correctly classifying points from different classes. Table ? illustrates a simple example where this is true for a variety of common evaluation metrics. We look at the values of the metrics on two related two-class problems. In both cases, the number of points in the positive and negative classes are both equal to 100. In case 1, the number of true positives is 80 while the number of true negatives is 50; in case 2, the opposite is true: the number of true positives is 50 while the number of true negatives is 80. Note that only accuracy is equal in both cases, while for all other metrics, there is clearly a different effect due to misclassifications in the positive versus the negative class. Several authors have noted that many evaluation metrics do not weight misclassification costs equally (e.g., (? )). In this paper, we introduce a framework for analyzing evaluation metrics that quantifies the impact of correctly classifying samples from each class. We then apply the framework to a number of common metrics. We show that our framework provides a new viewpoint on several known and previously unknown properties about metrics. Copyright c © 2007, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. Table 1: A simple example of asymmetry Metric Case 1 Case 2 accuracy 0.65 0.65 precision 0.62 0.71 recall 0.80 0.50 f1-measure 0.70 0.59 A note on notation: We will use lower-case letters for scalars and bold lower-case letters for functions. df dx denotes the derivative of a function f with respect to x while ∂g ∂x denotes the partial derivative of a function g with respect to x. For two-class problems, let constants n and n− represent the number of points in the positive and negative classes, respectively, and let n = n + n− represent the total number of data points being evaluated. Let ntp represent the current number of true positives and ntn represent the current number of true negatives. Similarly, nfp and nfn will refer to the number of false positives and false negatives, respectively; note that, nfn = n − ntp and nfp = n− − ntn, so any equations with ntp, ntn, nfp, and nfn can be written in terms of only ntp, ntn, and constants n, n, and n− for a given problem. For the multi-class case, k represents the number of classes, n represents the number of points from the jth class, njtp is the number of correctly classified points in class j and njfp is the number of points incorrectly classified as class j. Background on Evaluation Metrics A number of popular evaluation metrics have been introduced in the past for analyzing the predictions of classification algorithms. Below, we describe several common metrics analyzed in this paper. Total cost and variants If misclassification costs are known, a useful evaluationmetric is the total misclassification cost. Here, we discuss the case where there is a fixed misclassification cost for misclassifying a point from class i as a point from class j (? ). For a two-class problem, let the cost of making a false positive be denoted as cfp and similarly for true positives