Pattern Recognition Letters 30 (2009) 27–38

Contents lists available at ScienceDirect

Pattern Recognition Lettersjournal homepage: www.elsevier.com/locate/patrec

An experimental comparison of performance measures for classiﬁcation
C. Ferri *, J. Hernández-Orallo, R. Modroiu
Departament de Sistemes Informàtics i Computació, Universitat Politècnica de València, València 46022, Spaina r t i c l e

i n f oArticle history:
Received 18 December 2006
Received in revised form 4 December 2007
Available online 2 September 2008
Communicated by T.K. Ho
Keywords:
Classiﬁcation
Performance measures
Ranking
Calibrationa b s t r a c t
Performance metrics in classiﬁcation are fundamental in assessing the quality of learning methods andlearned models. However, many different measures have been deﬁned in the literature with the aim ofmaking better choices in general or for a speciﬁc application area. Choices made by one metric areclaimed to be different from choices made by other metrics. In this work, we analyse experimentallythe behaviour of 18 different performance metrics in several scenarios, identifying clusters and relationships between measures. We also perform a sensitivity analysis for all of them in terms of several traits:class threshold choice, separability/ranking quality, calibration performance and sensitivity to changes inprior class distribution. From the deﬁnitions and experiments, we make a comprehensive analysis of therelationships between metrics, and a taxonomy and arrangement of them according to the previous
traits. This can be useful for choosing the most adequate measure (or set of measures) for a speciﬁc application. Additionally, the study also highlights some niches in which new measures might be deﬁned andalso shows that some supposedly innovative measures make the same choices (or almost) as existing
ones. Finally, this work can also be used as a reference for comparing experimental results in pattern recognition and machine learning literature, when using different measures.
Ó 2008 Elsevier B.V. All rights reserved.

1. Introduction
The correct evaluation of learned models is one of the mostimportant issues in pattern recognition. One side of this evaluation
can be based on statistical signiﬁcance and conﬁdence intervals,
when we want to claim that one model is better than another orthat one method is better than another. The other side of evaluation relies on which metric is used to evaluate a learned model.
Evaluating a regression model with absolute error is certainly not
the same as doing so with squared error. In fact, for regression,
the relation and appropriateness of several evaluation measures
have been analysed both theoretically and experimentally (Harrell,
2001; Ripley, 1996; Bishop, 1995), and the difference between theexisting measures is sufﬁciently clear. The continuous character ofthe output (and measures) makes the task easier, especially on thetheoretical level. However, for classiﬁcation, there is a very extensive number of measures, some of them without a clearly justiﬁed
theoretical basis, some of them recently introduced, and there is nocomprehensive analysis of whether some of them bring a reallynew point of view when evaluating classiﬁers.
In this work, we concentrate on metrics for evaluating classiﬁers, such as Accuracy, F-measure, Rank Rate, Area Under the ROC
Curve (AUC), Squared Error (Brier Score), LogLoss/Entropy, etc.
* Corresponding author. Fax: +34 96 387 73 59.
E-mail addresses: cferri@dsic.upv.es (C. Ferri), jorallo@dsic.upv.es (J. HernándezOrallo), emodroiu@dsic.upv.es (R. Modroiu).
0167-8655/$ - see front matter Ó 2008 Elsevier B.V. All rights reserved.
doi:10.1016/j.patrec.2008.08.010

Some of these metrics have been introduced for very different
applications and, supposedly, measure quite different things. Morespeciﬁcally, we will use 18 different metrics, which we classify inthree families as follows: Metrics based on a threshold and a qualitative understanding oferror: accuracy, macro-averaged accuracy (arithmetic and geometric), mean F-measure (F-score) and Kappa statistic. These
measures are used when we want a model to minimise thenumber of errors. Hence, these metrics are usual in many direct
applications of classiﬁers. Inside this family, some of these measures are more appropriate for balanced or imbalanced datasets,
for signal or fault detection, or for information retrieval tasks.
 Metrics based on a probabilistic understanding of error, i.e. measuring the deviation from the true probability: mean absolute
error, mean squared error (Brier score), LogLoss (cross-entropy),
two versions of the probability (rank) rate and two measures forcalibration. These measures are especially useful when we want
an assessment of the reliability of the classiﬁers, not only measuring when they fail but whether they have selected the wrong
class with a high or low probability. This is also crucial for committee models (machine ensembles) to properly perform aweighted fusion of the models.
 Metrics based on how well the model ranks the examples: AUC(Flach et al., 2003), which for two classes is equivalent to theMann–Whitney–Wilcoxon statistic, and is closely related to theconcept of separability. These are important for many

28

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

applications, such as mailing campaign design, Customer Relationship Management (CRM), recommender systems, fraud detection,
spam ﬁltering, etc., where classiﬁers are used to select the best n
instances of a set of data or when good class separation is crucial.
We also include two other measures, SAUC and PAUC which aresupposedly in the middle between the last two groups.
In this paper, we analyse how these 18 metrics correlate to eachother, in order to ascertain to what extent and in which situations
the results obtained and the model choices performed with one
metric are extensible to the other metrics. The results show thatmost of these metrics really measure different things and in many
situations the choice made with one metric can be different fromthe choice made with another. These differences become larger
for multiclass problems, problems with very imbalanced class distribution and problems with small datasets. But it also shows that
other measures are highly correlated (or they make the same
choice) and results obtained on one measure could be extrapolated
to other measures. For instance, we will see that probabilistic rank
rate always makes the same decision as absolute error.
The analysis is completed with a set of experiments to quantify
sensitivity to four important traits which are present in some measures but not present in others. These traits are class thresholdchoice optimality, separability/ranking quality, calibration performance and sensitivity (or conversely robustness) to changes inprior class distribution. From this analysis, we can quantify the
relations on these ‘dimensions’, which is a very useful complement
to the results of the correlation analysis.
To our knowledge, this is the ﬁrst experimental work which thoroughly compares the most generally used classiﬁer evaluation metrics for binary and multiclass problems arriving at conclusions
regarding interdependence and sensitivity of these measures. It isalso the ﬁrst work which gives a comprehensive taxonomy of thesemeasures.
The paper is organised as follows: In the following Section, we
explore some related work. In Section 3, we introduce the measures employed in this work and we present a ﬁrst taxonomy,
based on their deﬁnition. Next, in Section 4, we explain the methodology used in the experiments. The results of these experiments
are detailed in Section 5. Section 6 presents four different experiments to analyse the sensitivity of the 18 measures to the four
above-mentioned traits: class threshold choice, separability/ranking quality, calibration performance and sensitivity to changes inprior class distribution. Finally, Section 7 includes the conclusions,and gives some ideas for future work.

2. Related works
Several works have shown the fact that usually, given a dataset,the learning method that obtains the best model according to agiven measure, is not the best method if we employ a different
measure. For instance, it is said in (Huang et al., 2003) that Naive
Bayes and pruned decision trees are very similar in predictive
accuracy. However, using exactly the same algorithms, in (Huang
and Ling, 2005) the authors show that Naive Bayes is signiﬁcantly
better than pruned decision trees in AUC. The different results cannot be explained here by slightly different implementations orvariants of machine learning algorithms, but on the fact that thetwo measures (accuracy and AUC) evaluate different things.
The relationship between AUC and accuracy has been speciallystudied. For instance, Cortes and Mohri (2003) makes a detailed
statistical analysis of the relationship between the AUC and the error rate. The results show that ‘‘the average AUC is monotonicallyincreasing as a function of the classiﬁcation accuracy, but that thestandard deviation for uneven distributions and higher error rates

is noticeable. Thus, algorithms designed to minimize the error rate
may not lead to the best possible AUC values”. On the other hand,
Rosset (2004) is a surprising work, since it shows that if we use
AUC for selecting models using a validation dataset, we obtain better results in accuracy (in a different test dataset) than whenemploying accuracy for selecting the models. Following this idea,(Wu et al., 2007) shows that an AUC-inspired measure (SAUC) is
better for selecting models when we want to improve the AUC ofthe models. It has also been shown Domingos and Provost(2003), Ferri et al. (2003) that although pruning usually improves
accuracy in decision trees, it normally decreases the AUC of thedecision trees. Speciﬁcally, most of the studies on the effect ofpruning on decision trees have been performed taking accuracy
into account (see e.g. Esposito et al. (1997)). Not many works canbe found which compare other measures in terms similar to thosein which accuracy and AUC have been studied recently. An exception is Davis and Goadrich (2006) where the authors study the relationship between ROC curves and Precision-Recall curves.
In Multi-Classiﬁer Systems (Kuncheva, 2004) many works havebeen devoted to study the resulting accuracy of an ensemble ofcombined classiﬁers given the original accuracies and some otherconditions (Melnik et al., 2004; Narasimhamurthy, 2005; Kuncheva
et al., 2003) or how to combine them in order to increase accuracy(Young Sohn, 1999). However, little is known when the performance measure is different from accuracy (Freund et al., 2003; Cortes and Mohri, 2003; Lebanon and Lafferty, 2002).
Finally, in other works Zadrozny and Elkan (2001), different
probabilistic measures (MSE, log-loss and proﬁt) are used to evaluate different methods for calibrating classiﬁers.
All of these works are difﬁcult to compare and understand together especially because there is no comprehensive study of theseveral metrics they are using to evaluate performance.
There are some previous works that compare some performancemeasures for classiﬁcation theoretically. Flach (2003), Fuernkranz
and Flach (2005) analyse several metrics (AUC, accuracy, Fmeasure)using the ROC space. Buja et al. (2005) studies several metrics (LogLoss, squared error, and others) checking whether these are properscoring rules, deﬁning proper score rules as, ‘‘functions that scoreprobability estimates in view of data in a Fisher-consistent manner”. Huang and Ling (2007) is also a theoretical work on the features a metric should have and proposing new ones.
However, empirical studies have been scarce and limited in literature. The only exception to this is Caruana and Niculescu-Mizil(2004), independent and simultaneous to a preliminary work ofours (Ferri et al., 2004). Caruana and Niculescu-Mizil’s work analyses the behaviour of several performance measures against agreat number of machine learning techniques, the relationship between the measures using multi-dimensional scaling and correlations and, ﬁnally, derives a new central measure based on othermeasures. The main goal of the paper is to analyse which familyof algorithms behaves best with which family of measures. We disagree on this point because each machine learning family of algorithms includes hundreds or even thousand of variants. Some ofthem are tuned to optimise accuracy, others to optimise MSE, others to optimise AUC, . . . so a clear result on whether neural networks, or support-vector machines are better for this or othermeasures is, in our opinion, very difﬁcult to state.
Additionally, the work in (Caruana and Niculescu-Mizil, 2004)
only used two-class measures and relatively large datasets. Smalldatasets are essential, because of the size of the training and thetest set is an especially important issue when comparing measures: measures based on a probabilistic view of error integrate
more information than qualitative ones and, consequently, they
are supposedly better for smaller datasets.
The methodology used in (Caruana and Niculescu-Mizil, 2004)’s
analysis is different. Some experiments use Euclidean distances

29

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

between measures, which requires normalisation and, even with
this, it is very sensitive to outliers, non-linear behaviours (e.g.
MSE) or unboundness (e.g. LogLoss). Then they use correlations.
They combine multi-dimensional scaling (on a 2D projection) withthe results from the correlation matrix, but they do not use any
clustering techniques to arrange the measures. As we will see below, there are more than two basic dimensions, so we decided
not to use projections (as multi-dimensional scaling) as a result.
Finally, our work is much more exhaustive in many ways. We
include many more measures, some of them very important, suchas macro-averaged accuracy, the AUC variants and probability rate,
we use a much larger number of datasets, we analyse the results ina segmented way for two-class vs. multiclass, small vs. large datasets, balanced vs. imbalanced, etc.
After these differences in goals and methodologies, the results
obtained by Caruana and Niculescu-Mizil (2004) and the rest ofworks mentioned in this section, as we will see below, are more
complementary than overlapping with this work.

3. Measures
In this section, we present the deﬁnition of the measures wewill analyse. The selection of measures is based both on the properties of each measure (we want to cover a broad range of different
measures) and their use (we want to cover the most popular ones
in machine learning and pattern recognition literature).
Taking into account a broad range of different measures, we areparticularly interested in three types of measures, as mentioned in
the introduction: measures that are sensitive to a good choice ofthreshold, measures that quantify the quality of rankings (separability), and measures which quantify the deviation of the estimatedprobability wrt. to the actual probability. For instance, a very goodclassiﬁer in terms of separability (rankings) can yield very bad
accuracy if we choose a bad threshold to separate the classes. On
the other hand, a classiﬁer can have very good results for a threshold, but perform very badly for other thresholds (when costs orcontext changes, as ROC analysis deals with).
The difference which is sometimes most difﬁcult to grasp isthe difference between good rankings and good probabilities. A
classiﬁer can produce very good rankings, but probabilities might
differ from the actual probabilities. In this case, we say that theclassiﬁer is not well calibrated. More precisely, calibration is deﬁned as the degree of approximation of the predicted probabilities to the actual probabilities. It is usually a measure of thereliability of the prediction (DeGroot and Fienberg, 1982). If we
predict that we are 99% sure, we should expect to be right 99%of the times. More formally, a classiﬁer is perfectly calibrated if
for a sample of examples with predicted probability p, the expected proportion of positives is close to p. The problem of measuring calibration is that the test set must be split into several
segments or bins. If too few bins are deﬁned, the real probabilities
are not properly detailed to give an accurate evaluation. If too
many bins are deﬁned, the real probabilities are not properly estimated. A partial solution to this problem is to make the bins overlap. These different approaches have produced several measures
to estimate calibration.
In fact, the relation between good class separability and calibration has been analysed in literature. The most remarkable approach is based on the so-called ‘‘decompositions of the Brierscore” (Sanders, 1963; Murphy, 1972), which separate the Brierscore (Mean Squared Error) measure into a reliability, a resolution,
and an uncertainty components, or, alternatively, into a calibration
term and reﬁnement term. This calibration term requires binning.
Below we will introduce the deﬁnitions of 18 measures. The
ﬁrst 5 are qualitative and the other 13 are probabilistic.

3.1. Deﬁnition of measures
We use the following notation. Given a (test) dataset, m denotes
the number of examples, and c the number of classes. f ði; jÞ represents the actual probability of example i to be of class j. We assume
that f ði; jÞ always takes values in {0, 1} and is strictly not a probaPmbility but an indicator function. With mj ¼ i¼1 f ði; jÞ, we denotethe number of examples of class j. pðjÞ denotes the prior probabilityof class j, i.e., pðjÞ ¼ mj =m.
Given a classiﬁer, pði; jÞ represents the estimated probability ofexample i to be of class j taking values in [0, 1]. C h ði; jÞ is 1 iff j is thepredicted class for i obtained from pði; jÞ using a given threshold h(or decision rule, especially in multiclass problems). Otherwise,C h ði; jÞ is 0. We will omit h below.
 Accuracy: (Acc). This is the most common and simplest measure
to evaluate a classiﬁer. It is just deﬁned as the degree of right
predictions of a model (or conversely, the percentage of misclassiﬁcation errors)

Pm Pc

j¼1 f ði; jÞCði; jÞ

i¼1

Acc ¼

m

:

 Kappa statistic: (KapS). This is originally a measure of agreement between two classiﬁers (Cohen, 1960), although it can also
be employed as a classiﬁer performance measure (Witten andFrank, 2005) or for estimating the similarity between the members of an ensemble in Multi-classiﬁers Systems (Kuncheva,
2004)

KapS ¼PðAÞ  PðEÞ;1  PðEÞ

where PðAÞ is the relative observed agreement among classiﬁers,
and PðEÞ is the probability that agreement is due to chance. In
this case, PðAÞ is just the accuracy of the classiﬁer, i.e.
PðAÞ ¼ Acc as deﬁned above, and PðBÞ is deﬁned as follows:Pc hPc Pm
k¼1PðEÞ ¼j¼1

i hP Picm
j¼1
i¼1 f ði; jÞCði; kÞ

i¼1 f ði; kÞCði; jÞm2:

 Mean F-measure: (MFM). This measure has been widely
employed in information retrieval (Baeza-Yates and RibeiroNeto, 1999)

F  measure ðjÞ ¼

2  recall ðjÞ  precision ðjÞ;recall ðjÞ þ precision ðjÞ

wheremcorrectly classified positives X
f ði; jÞCði; jÞ;¼
total positives
mj
i¼1
Pm
correctly classified positivesf ði; jÞCði; jÞ¼ i¼1;
precision ðjÞ ¼
Pmj
total predicted as positives
i¼1 Cði; jÞrecall ðjÞ ¼

where j is the index of the class considered as ‘‘positive”. Finally,mean F-measure is deﬁned as follows:

Pc
MFM ¼

j¼1 F  Measure ðjÞc: Macro average arithmetic: (MAvA). This is deﬁned as the arithmetic average of the partial accuracies of each class. This is usually referred as macro average (Mitchell, 1997).

Pc
MAvA ¼

j¼1

Pm
i¼1

f ði;jÞCði;jÞ
mj

c

:

 Macro average geometric: (MAvG). This is deﬁned as the geometric average of the partial accuracies of each class.

30

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

vﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
uY
c Pm
u
c
i¼1 f ði; jÞCði; jÞ:
MAvG ¼ t
mj
j¼1
 AUC of each class against the rest, using the uniform class distribution: (AUNU). The AUC (Area Under the ROC Curve) (Fawcett, 2006) of a binary classiﬁer is equivalent to the probability
that the classiﬁer will rank a randomly chosen positive instancehigher than a randomly chosen negative instance (Mann–Whitney–Wilcoxon statistic interpretation).

Pm
AUC ðj; kÞ ¼

i¼1 f ði; jÞPm

t¼1 f ðt; kÞIðpði; jÞ; pðt; jÞÞmj  mk:IðÞ is a comparison function satisfying Iða; bÞ ¼ 1 iff
a > b; Iða; bÞ ¼ 0 iff a < b and Iða; bÞ ¼ 0:5 iff a ¼ b.
This measure has been extended for multi-class problems in
more than one way. We present four variants below.
AUNU computes the area under the ROC curve treating a c-dimensional classiﬁer as c two-dimensional classiﬁers, where classes are
assumed to have uniform distribution, in order to have a measure
which is independent of class distribution change. Formally

Pc
AUNU ¼

j¼1 AUC ðj; restj Þc;where restj gathers together all classes different from class j.
Here, the area under the ROC curve is computed in the one
against all approach, i.e. we compute this measure as the averageof c combinations.
 AUC of each class against the rest, using the a priori class distribution: (AUNP). This measure (Fawcett, 2001) computes thearea under the ROC curve treating a c-dimensional classiﬁer asc two-dimensional classiﬁers, taking into account the priorprobability of each class ðpðjÞÞAUNP ¼

c
Xwhich is equal to AUC ði; jÞ except from an additional factor
ðpði; jÞ  pðt; kÞÞ, which is added to quantify the deviation in probability estimation whenever the rank is incorrect.
From the binary AUC, the multiclass SAUC measure is deﬁned as

SAUC ¼

c X
cX
1Scored AUC ðj; kÞ:cðc  1Þ j¼1 k–j

 Probabilistic AUC: (PAUC). is also a variant of the AUC which
includes probabilities in the deﬁnition (Ferri et al., 2004) in moreor less the same line as SAUC, although the part with the indicator function is no longer used. This means that it is not a properrank measure.First, Probabilistic AUC for two classes is deﬁned
as

Pm

i¼1

Prob AUC ðj; kÞ ¼:

PAUC ¼

c X
cX
1
Prob AUC ðj; kÞ:cðc  1Þ j¼1 k–j

 Macro Average Mean Probability Rate: (MAPR). It is computed
as an arithmetic average of the mean predictions for each class(see Mitchell, 1997)Pc
MAPR ¼

Pm

i¼1

j¼1

f ði;jÞpði;jÞ
mj

c

:

 Mean Probability Rate: (MPR). This measure is also a measure
which analyses the deviation from the true probability. It is a
non-stratiﬁed version of the previous one, the arithmetic average of the predicted probabilities of the actual class (Lebanon
and Lafferty, 2002)

Pc PmpðjÞAUC ðj; restj Þ: AUC of each class against each other, using the uniform class
distribution: (AU1U). This metric also represents the approximation of AUC in the case of multi-dimensional classiﬁers, computing the AUC of cðc  1Þ binary classiﬁers (all possible
pairwise combinations) and considering uniform distributionof the classes (Hand and Till, 2001)
c X
c
X
1
AUC ðj; kÞ:cðc  1Þ j¼1 k–j

 AUC of each class against each other, using the a priori class
distribution: (AU1P). In order to complete all the reasonable
extensions of AUC for more than two classes, we deﬁne a ﬁnal
AUC-based measure. This measure represents the approximation of AUC in the case of multi-dimensional classiﬁers, computing AUC of cðc  1Þ binary classiﬁers and considering the a priori
distribution of the classes

AU1P ¼

2

From the binary AUC, the multiclass PAUC measure is deﬁned as

MPR ¼

j¼1

i¼1 f ði; jÞpði; jÞm

j¼1

AU1U ¼

m
P
f ði;jÞpði;jÞ
f ði;kÞpði;jÞ

þ1
mj
mk
i¼1

c X
c
X
1
pðjÞAUC ðj; kÞ:cðc  1Þ j¼1 k–j

 Scored AUC: (SAUC). is a variant of the AUC, which includes
probabilities in the deﬁnition (Wu et al., 2007). The idea is tointroduce a rank measure which is robust to rank changes due
to small probability variations.
First, Scored AUC for two classes is deﬁned as

Scored AUCðj; kÞPPmf ði; jÞ m
t¼1 f ðt; kÞIðpði; jÞ; pðt; jÞÞ  ðpði; jÞ  pðt; kÞÞ¼ i¼1;
mj  mk: Mean Absolute Error: (MAE). This metric shows how much thepredictions deviate from the true probability and it only differsfrom the previous one in that the product is changed by theabsolute value of the difference

Pc Pm
MAE ¼

j¼1

i¼1 jf ði; jÞ  pði; jÞjmc: Mean Squared Error: (MSE). This is just a quadratic version ofMAE, which penalises strong deviations from the true probability. This metric is also known as Brier score (Brier, 1950) andintegrates calibration and other components, usually grouped
under the term ‘reﬁnement’

Pc Pm
MSE ¼

j¼1

2
i¼1 ðf ði; jÞ  pði; jÞÞmc: LogLoss: (LogL). This is also a measure of how good probability
estimates are (also known as cross entropy) and it has been used
when calibration is important (Good, 1952, 1968; Dowe et al.,
1996)

P P
 cj¼1 m
i¼1 ðf ði; jÞlog2 pði; jÞÞLogL ¼:
m
To avoid the case of log2 ð0Þ; log2 pði; jÞ is computed aslog2 ðmaxðpði; jÞ; ÞÞ, where  will be set to 0.00001 for theexperiments.
 Calibration Loss: (CalL). In (Fawcett and Niculescu-Mizil, 2007)
and, independently, in (Flach and Takashi Matsubara, 2007), therelationship between the AUC-based measures, and ROC analysis in general, with calibration has been clariﬁed. A perfectly
calibrated classiﬁer always gives a convex ROC curve. A method

31

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

for calibrating a classiﬁer is to compute the convex hull or,equivalently, to use isotonic regression. In (Flach and Takashi
Matsubara, 2007), they derive a decomposition of the Brier Score
into calibration loss and reﬁnement loss. Calibration loss isdeﬁned as the mean squared deviation from empirical probabilities derived from slope of ROC segments

CalLoss ðjÞ ¼

rj
X
X

0
@pði; jÞ b¼1 i2sj;b

X f ði; jÞ
i2sj;b

jsj;b j12
A ;where rj is the number of segments in the ROC curve for class j,i.e. the number of different estimated probabilities for classj : jfpði; jÞgj. Each ROC segment is denoted by sj;b , with b 2 1::r j ,and formally deﬁned assj;b ¼ fi 2 1;...;mj 8k 2 1;...;m : pði;jÞ P pðk;jÞ ^ i R sj;d ; 8d < bg:
From the previous binary CalLoss, the general multiclass Calibration Loss measure is deﬁned as

CalL ¼c
1X
CalLoss ðjÞ:c j¼1

 Calibration by Bins: (CalB). A calibration measure based on
overlapping binning is CAL (Caruana and Niculescu-Mizil,
2004). This is deﬁned as follows. For each class, we must order
all cases by predicted positive class pði; jÞ, giving new indices
i . Take the 100 ﬁrst elements (i from 1 to 100) as the ﬁrst
bin. Calculate the percentage of positives (class j) in this bin as
the actual probability, ^f j . The error for this bin isP^i 21;...;100 jpði; jÞ  f j j. Take the second bin with elements from 2
to 101 and compute the error in the same way. At the end,
average the errors. The problem of using 100 as (Caruana andNiculescu-Mizil, 2004) suggest is that it might be a much toolarge bin for small datasets. Instead of 100 we set a different bin
length, s ¼ m=10, to make it more size-independent. Formally:CALðjÞ ¼Pbþs1  ms bþs1
X  
1 X
i ¼b f ði ; jÞ
pði
;
jÞ

:


m  s b¼1 i ¼b s
We indicate with i that indices are ordered by pði; jÞ. For morethan two classes, the measure is the average for all classes, i.e.

CalB ¼c
1XCALðjÞ:c j¼1

Some of the previous measures (MFM, MAvA, MAvG, AUC variants, SAUC, PAUC, MAPR, MPR, CalL and CalB) have to be carefully implemented to exclude any class for which the test set
has no instances.

Table 1
Characterisation of measures according to different traits
MeasureClass threshold

Calibration

Ranking

Class frequencies

Acc
KapS
FME
MAVA
MAVG
AU1u
AU1p
AUnu
AU1p
SAUC
PAUC
MAPR
MPR
MAE
MSE
LogL
CalL
CalB

Yes
Yes
Yes
Yes
Yes
No
No
No
No
No
No
No
No
No
No
No
No
No

No
No
No
No
No
No
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes

No
No
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes

Yes
Yes
Partially
No
No
No
Yes
No
Yes
No
No
No
Yes
Yes
Yes
Yes
No
Yes

thresholds, changes in calibration which preserve the ranking,changes in ranking which do not cross the class thresholds (butusually affect calibration), and changes in class frequency.
As can be seen in the table, according to the ﬁrst three traits,
threshold, calibration and ranking, the measures can be grouped
as those focused on error (yes, no, no), those focused on ranking(no, no, yes) and those focused on probabilities (no, yes, yes). The
fourth trait, sensitivity to class frequency change is present in someof them.
Some interesting things can be observed from the table. The
most surprising issue is that there is no measure which has a‘Yes’ in both the class threshold column and either the calibration
or ranking columns. This means that, to date, and as far as weknow, there is no measure which simultaneously combines thethreshold and the estimated probability. In fact, in the deﬁnitionof the 18 measures, none of them use Cði; jÞ and pði; jÞ at the sametime. This would be a good niche to study in the future, especiallybecause in some applications the deviation from the actual probability is only relevant when the classiﬁer fails. For instance, measures with a term like f ði; jÞ  Cði; kÞ  pði; kÞ might be analysed.
Another interesting observation from the table is that many measures have exactly the same characterisation for the four traits,
so the differences can only be shown from a quantitative analysis.
For instance, it seems that LogL is more sensitive to calibration
than MSE, and MSE is more sensitive than MAE, but we cannot
quantify these difference from the previous table. The experimental analyses in Sections 5 and 6 will conﬁrm one of these statements and refute the other.

3.2. Taxonomy of measures according to their properties
4. Methodology
Previously, we mentioned that from the 18 measures, the ﬁrst 5
measures are qualitative and the remaining 13 are probabilistic.
This is now clear from the deﬁnitions if we just check that the ﬁrst
5 use the term Cði; jÞ in their deﬁnition (which is compared to theactual f ði; jÞ). So, the ﬁrst 5 measures are sensitive to the classthreshold. The other 13 measures do not use the term Cði; jÞ in theirdeﬁnition but use the term pði; jÞ. This is the estimated probability
which is compared to the actual probability f ði; jÞ.
We can enrich the previous analysis if we also consider whether
the measure takes into account the ranking (this corresponds to
the Iðpði; jÞ; pðt; jÞÞ term) but not the direct value of the probability
estimation. Additionally, we can also analyse whether the measures are sensitive to class frequency changes or not.
In Table 1, we indicate whether each of the 18 measures is inﬂuenced or not by changes in these four traits: changes in class

The experiments were performed using Witten and Frank(2005), which we extended with several new metrics, not includedin the current distribution. We used six well-known machinelearning algorithms: J48, Naive Bayes, Logistic Regression, Multilayer Perceptron, K-Nearest Neighbour, AdaBoost with ten J48
trees and we performed the experiments with 30 small and medium-size datasets included in the machine learning repository(Blake and Merz, 1998), 15 of them being two-class (binary) problems and 15 of them being multiclass. Also half of them are considered to be balanced, and the rest, imbalanced. Table 2 includes
further details of the datasets (size, number of classes, number of
nominal attributes, number of numerical attributes, percentageof the majority class). The datasets which we consider balanced
are highlighted in bold.

32

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

Table 2
Datasets used in the experiments#

Datasets

Size

Classes

Nom.

Num.

%Maj–%Min.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

Autos5c
Balance Scale
Breast Cancer
Chess
Cmc
Credit rating
Dermatology
German–credit
Glass
Heart–statlog
Hepatitis
House voting
Ionosphere
Iris plan
Monks1
Monks2
Monks3
New thyroid
Pima
Sick
SoybeanSegmentation
Spect
Tae
Tic–tac
Vehicle3c
Waveform
Wine
Wovel
Zoo

202
625
286
3196
1473
690
366
1000
214
270
155
435
351
158
556
601
554
215
768
3772
683
2310
80
151
958
846
5000
178
990
101

5
3
2
2
3
2
6
2
6
2
2
2
2
3
2
2
2
3
2
2
19
7
2
3
2
3
3
3
11
7

10
0
0
36
7
9
33
13
9
13
14
16
0
0
6
6
6
0
0
22
31
0
0
2
8
0
0
0
3
16

15
4
9
0
2
6
1
7
0
0
5
0
34
4
0
0
0
5
8
7
0
19
44
3
0
18
21
13
11
1

33.16–10.89
46.08–7.84
70.27–29.72
52.22–47.48
42.70–22.61
55.50–44.50
30.60–5.46
70.00–30
35.51–4.2
55.55–44.45
79.35–20.65
54.25–45.75
64.10–35.9
33.33–33.33
50–50
65.72–34.27
51.99–48.01
69.97–13.95
65.10–34.90
93.87–6.12
13.46–1.17
14.28–14.28
50–50
34.43–32.45
65.34–34.65
51.41–23.52
33.92–33.06
39.88–26.97
9.09–9.09
40.6–3.96

The above mentioned models were evaluated using 20  5 foldcross-validation, each of the 6 models being applied to each of the
30 datasets, getting 600 results for each dataset, making 18,000 results in total. We set up seven types of analysis: an overall analysis
for all datasets, an analysis for binary and multiclass problems, forbalanced and imbalanced problems, and for short datasets andlarge datasets. In each case we calculated the Pearson (standard)linear correlation and Spearman rank correlation between all eighteen metrics. Apart from the global view for all the datasets which
will be presented using linear and rank correlations, the other six
analyses will only be shown for rank correlations.
It is important to remark that we compute the correlation foreach dataset, i.e., we analyse the results of the 6 models for one
dataset and the corresponding 100 combinations of the cross-validation. Not merging results from different datasets is crucial, sincemeasure values are inﬂuenced in very different ways depending on
the dataset, e.g. number of classes, imbalance, problem difﬁculty,
etc. Consequently, we construct one correlation matrix per dataset.
Finally, we average (arithmetically) the 30 correlation matrices.
The results when we use the Pearson correlation indicate thestrength and direction of a linear relationship between two measures, while the rank correlation assesses how well an arbitrary
monotonic function could describe the relationship between two
variables, without making any assumptions about the frequency
distribution of the variables. A high rank correlation between
two measures means that for the same problem these two measures have ranked the 6 models similarly. In other words, bothmeasures would usually select the same model. Unlike the standard correlation coefﬁcient, the assumption that the relationshipbetween the variables is linear is not required. In order to avoid
negative values for correlation we worked with 1-MAE, 1-MSE, 1LogL, 1-CalL and 1-CalB.
Since there are hence eight correlation matrices, and these are
difﬁcult to understand at a glance, we will use dendrograms forrepresentation; where the linkage distance is deﬁned asð1  correlationÞ. A dendrogram is a tree diagram frequently used
to illustrate the arrangement of the clusters produced by a clustering algorithm. This kind of diagram has several advantages: we caneasily visualise the clusters formed by the measures, as well as thelinkage distance among clusters. It is also quite easy to ﬁnd out the
number of clusters and their components once we have selected alinkage distance. There are several methods for constructing a dendrogram using a linkage distance. We will use the ‘‘average group
distance” method, which joins an existing group to the element (orgroup) whose average distance to the group is minimum.

5. Analysis of results
In this section, we discuss some of the interesting outcomes wefound from the analysis of the correlation between metrics. First
we analyse the correlation matrix (both linear and rank) for all
datasets, as shown in Table 3.
In Fig. 1, we show dendrograms built from the obtained linear
and rank correlations using all the available results. This ﬁgure represents the relations between the measures in an abridged andmore comprehensible way.
The correlations shown on the matrix for both kinds of correlations, as well as both dendrograms are very similar (rank correlations are slightly higher than linear correlations, as expected).
Consequently, there is no point in replicating the analysis. Hence,
we will focus on the results and dendrogram for rank correlation.
A general observation is that all correlations are positive, andusually strong (greater than 0.5). The only clear exceptions aresome correlations between LogL and some probabilistic measures,
which we might consider in the same family a priori. This is mostly
due to the fact that LogL is an unbounded measure, i.e, a 0 wrong
probability has an inﬁnity penalty (or very high if logð0Þ is avoided
in implementations).
A ﬁrst speciﬁc observation is the close relationship between all
the qualitative measures: MAvA and MAvG, MFM, Acc and KapS, as
expected. Although not exactly equal, their choices are almost thesame. A second speciﬁc observation can be made with the ranking
measures (AU*). The 4 variants of AUC behave quite similarly, sothey can even be used interchangeably. This means that previous
studies in literature using these different variants for evaluating
rankers can be contrasted safely, independently of which variant
they have used. Additionally, it is interesting to note that no othermeasure correlates to AUC more than 0.82, justifying the use of the
AUC as a genuinely different and compact measure/family. Finally,on probabilistic measures, there is a clear equivalence between
MPR and MAE, which is not surprising if we take a look at their definitions. The same happens for PAUC and MAPR. In fact, it is shownthat PAUC has no relation whatsoever with AUC. These four measures, jointly with SAUC collapse at a linkage distance of 0.1, whichmeans that all of them are very similar. MSE behaves differently, as
does the fore-mentioned LogL, which seem to be out of this group.
In fact, if we take a look to dendrograms, and using linkage distance 0.1, we discover 7 clusters: AUC measures, qualitative measures, ‘plain’ probabilistic measures (MPR, MAE, PAUC, MAPR)
with SAUC, and then 4 isolated measures: MSE, LogL, CalL and CalB.
MSE and LogL use a quadratic or logarithmic function on the probabilities, which might explain their distance to the other probabilistic measures. The two calibration measures are outsiders
because they are not proper performance measures and they try
to recognise the degree in which probabilities are calibrated. Theirclosest measure is MSE (correlations about 0.7), since MSE can be
decomposed into a calibration term and other terms. However, it
is interesting to see that neither calibration measures correlate
well (0.42), which suggest that both measures of calibration are
signiﬁcantly different (partly because bins for CalL are smaller).

33

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38
Table 3
Linear (bottom-left) and rank (top-right) correlation results for all datasets
–

Acc

Acc
KapS
MFM
MAvA
MAvG
AU1U
AU1P
AUNU
AUNP
SAUC
PAUC
MAPR
MPR
MAE
MSE
LogL
CalL
CalB

0.98
0.95
0.90
0.85
0.69
0.71
0.69
0.71
0.67
0.74
0.74
0.78
0.78
0.88
0.47
0.61
0.61

KapS

MFM

MavA

MavG

A1U

A1P

ANU

ANP

Sauc

Pauc

Mapr

MPR

MAE

MSE

LogL

CalL

CalB

0.98

0.95
0.97

0.92
0.95
0.98

0.83
0.88
0.93
0.95

0.70
0.72
0.71
0.73
0.67

0.72
0.74
0.71
0.71
0.66
0.99

0.70
0.72
0.70
0.71
0.66
1.00
0.99

0.72
0.73
0.69
0.70
0.64
0.97
1.00
0.98

0.68
0.71
0.74
0.75
0.75
0.47
0.46
0.45
0.45

0.76
0.79
0.81
0.81
0.80
0.58
0.57
0.56
0.55
0.97

0.76
0.79
0.81
0.81
0.80
0.58
0.57
0.56
0.55
0.97
1.00

0.79
0.78
0.78
0.74
0.72
0.52
0.52
0.50
0.51
0.92
0.95
0.95

0.79
0.78
0.78
0.74
0.72
0.52
0.52
0.50
0.51
0.92
0.95
0.95
1.00

0.88
0.87
0.85
0.82
0.75
0.81
0.82
0.80
0.81
0.61
0.72
0.72
0.73
0.73

0.40
0.40
0.38
0.38
0.32
0.67
0.67
0.67
0.67
0.03
0.14
0.14
0.11
0.11
0.63

0.55
0.55
0.54
0.51
0.47
0.45
0.46
0.46
0.46
0.32
0.40
0.40
0.42
0.42
0.67
0.55

0.58
0.56
0.57
0.52
0.52
0.48
0.48
0.46
0.46
0.60
0.65
0.65
0.69
0.69
0.66
0.24
0.29

0.97
0.94
0.90
0.71
0.73
0.71
0.73
0.71
0.77
0.77
0.77
0.77
0.87
0.47
0.61
0.59

0.97
0.95
0.70
0.70
0.70
0.69
0.73
0.79
0.79
0.77
0.77
0.85
0.45
0.59
0.59

0.98
0.72
0.71
0.71
0.70
0.74
0.80
0.80
0.72
0.72
0.81
0.45
0.55
0.55

0.69
0.68
0.68
0.67
0.75
0.80
0.80
0.70
0.70
0.76
0.42
0.53
0.53

0.98
1.00
0.97
0.45
0.55
0.55
0.48
0.48
0.80
0.73
0.47
0.49

0.99
0.99
0.45
0.55
0.55
0.49
0.49
0.81
0.74
0.48
0.50

0.98
0.43
0.54
0.54
0.47
0.47
0.79
0.73
0.47
0.48

0.43
0.53
0.53
0.48
0.48
0.81
0.73
0.48
0.48

0.97
0.97
0.90
0.90
0.58
0.08
0.38
0.57

1.00
0.93
0.93
0.70
0.20
0.46
0.64

0.93
0.93
0.70
0.20
0.46
0.64

1.00
0.71
0.17
0.50
0.67

0.71
0.17
0.50
0.67

0.67
0.70
0.69

0.50
0.31

0.42

Fig. 1. Dendrograms of standard correlations (left) and rank correlations (right) between the metrics for all datasets.

If we use linkage distance 0.3, we discover 4 clusters. One with the
AUC measures and LogL, a second cluster with all the probabilistic
and qualitative measures (including MSE), and a third and fourth
isolated measures: CalL and CalB. It is remarkable that MSE ﬁnds
its highest correlation with the qualitative measures (Acc in particular) and LogL with the AUC measures. Other surprising results are
the low correlation between the AUC measures and some of the derived measures (SAUC and PAUC), especially SAUC, which has an
AUC-based deﬁnition, but only shows correlations of about 0.45.
LogL shows the worst correlation of all with the group of ‘plain’probabilistic measures (MPR, MAE, PAUC, MAPR, SAUC), mainly
due to its logarithmic behaviour.
Despite the methodology being different, these results are consistent with (Caruana and Niculescu-Mizil, 2004), our previous preliminary results (Ferri et al., 2004) and other works we have referred
to in Section 2, the only difference being that we do not ﬁnd a strong
correlation between MSE and LogL, which was found on these twoworks, and might be found in the implementation of LogL, whichmight avoid logð0Þ in different ways and also because (Caruana
and Niculescu-Mizil, 2004) only analyses two class problems.
In fact, if we compare the correlation results of 2-class problems
with multiclass problems (see Fig. 2), we get some expected results. All the AUC variants collapse for 2 classes, since they are
all extensions of multiclass problems but equivalent to 2-classproblems. The rest of the correlations are similar in both cases

although a little bit stronger for multiclass problems. The only
big difference is that MSE is joined to the AUC measures in the
2-class datasets and not to the qualitative measures. This suggeststhat MSE behaves differently for 2-class problems and multiclassproblems.
If we compare the correlations for the datasets with balanced
class distribution against the correlations for the datasets withimbalanced class distribution (see Fig. 3), the results show morevariations. Correlations are much lower for imbalanced datasets,
and the way in which the deﬁnition of each measure mixes the partial functions for each class is very relevant. For instance, qualitative
measures are very close for balanced datasets. There is virtually no
difference between Acc, KapS, MAvG, MFM and MAvA for balanced
datasets, while it is ampliﬁed for imbalanced dataset. In fact, CalL is
associated with qualitative measures for balanced datasets, but notfor imbalanced datasets. The same thing occurs for LogL, which is
associated with AUC measures for balanced datasets, but not for
imbalanced datasets. Finally, it happens conversely with CalB. It is
associated with ‘plain’ probabilistic measures for imbalanced datasets, but not for balanced datasets. This highlights the relevance of
metric choice depending on the class balance of the dataset.
Finally, the results for relatively small vs. large datasets (seeFig. 4), is signiﬁcant around linkage distance of 0.2. While qualitative measures and AUC measures are joined for small datasets, and
it is the probabilistic measures which gather more information

34

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

Fig. 2. Dendrograms of rank correlations between the metrics for two-class datasets (left) and multi-class datasets (right).

Fig. 3. Dendrogram of rank correlations between the metrics for balanced datasets (left) and imbalanced datasets (right).

from less data, when it comes to large datasets, AUC and LogL arevery clearly separated from the rest of the measures.
6. Sensitivity analysis
The previous analysis disentangles the relationships of the performance measures and clusters them in groups, according to theircorrelation. From the deﬁnitions, the correlations and the clusters,
we have been able to give an interpretation of these groups andtheir relation to the three families of measures: those based on error, those based on ranking and those based on probabilities. Additionally, we have also analysed the inﬂuence of imbalance to thesemeasures. However, for those cases where we have a ‘‘Yes” on one
trait we do not know the degree of sensitivity to that trait. The following experiments try to shed more light on this. Furthermore,some measures integrate features from more than one of the previous families and behave differently to changes on prior probability distribution (especially, class frequencies). To gain more insight
into the relationship of these measures and their ability to capture
misclassiﬁcations, bad rankings (separability), bad probabilities,
and class proportion drifts, we have devised some experiments
to directly assess these issues, to complement the theoreticalarrangement performed in Table 1.

We present four experiments on model selection over four synthetic scenarios. The idea is that, given two models M1 and M 2 ,with M1 being better than M 2 , we progressively introduce somenoise to both models to check whether the performance measures
are able to choose M1 . In order to analyse the four traits mentioned
above, noise is applied to both models in four different ways.
 Misclassiﬁcation noise: Noise is applied to actual classes. In this
scenario we measure how sensitive the measure is to changes
in the actual class produced.
 Probability noise: Noise is applied to the probabilities the models
produce for each prediction. In this scenario, we measure how
sensitive the measure is to changes on model probabilities. Thiscan be interpreted as a situation where we analyse the reliabilityof the estimated probabilities, good calibration, etc.
 Ranking noise: Noise is applied to the ranking of the model prediction. In this scenario we measure how sensitive the measure
is to model ranking change. This can be interpreted as situations
where we analyse the reliability of the order of scores, i.e. goodor bad class separability, etc.
 Class proportion noise: Noise is applied to the frequency of thedataset classes, i.e. we vary the proportion of classes. In this scenario we measure how sensitive the measure is to class

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

35

Fig. 4. Dendrogram of rank correlations between the metrics for small datasets (left) and large datasets (right).

proportion drifts. This can be interpreted as situations where weanalyse the robustness (or conversely, sensitivity) to changes inprior class distribution.
The ﬁrst part for the four experiments is the same. Two binary(two-class) models are created. These artiﬁcial classiﬁers are randomly created in the following way. First, we generate 100 real
numbers from a uniform distribution on [0, 1]. These numbers represent probabilities of the positive class. We order these probabilities decreasingly. Secondly, we assign a positive class to theelements where the probability is greater than 0.5 and negativeclass to the rest. Finally, we randomly modify 10 probabilities
using the same uniform distribution on [0, 1]. As a result, we havea classiﬁer, denoted by M 1 , with a good separability/ranking and aperformance level of about 95%. This is just a way to generate suchan artiﬁcial classiﬁer, and does not have a signiﬁcant inﬂuence on
the following experiments if done differently (provided that theclassiﬁer has good separability and classiﬁcation rate).
The second classiﬁer, M2 , is obtained from the ﬁrst one by randomly modifying 10 additional probabilities (different from theones which were modiﬁed on the ﬁrst classiﬁer and copied tothe second). Consequently, on average, the second classiﬁer has
worse separability and a worse classiﬁcation rate (about 90%) thanthe ﬁrst one.
A remarkable issue of this setting is that, as a result, M 2 is better
calibrated initially than M1 . We will see this through the CalB measure, which will be ignored during the analysis, due to its ‘strange’
behaviour, since it is not a performance metric useful for model
selection, but is purely a measure of calibration.
We generate two classiﬁers in the previous way 10.000 times(so giving 10.000 experiments) for each level of noise. For eachexperiment we record the selection made by the measure. If it selects M1 , we score 1, if it select M 2 , we score 0. For ties, we score0.5. When no noise is introduced, since M1 is generally better thanM2 , all performance measures should select M1 on average. Thingschange, though, if we introduce some type of noise to M1 and M 2 .
The following experiments gradually introduce different typesof noise, in order to check the degree to which the selection madeby each measure is affected.

label is modiﬁed on the dataset) to 100% (where all class labels
are randomly generated). At 100% noise, all labels are new andboth models should behave similarly. In the rest, since M 1 must
be better than M2 , we record the estimated probability (i.e. frequency) of making a wrong guess (choosing M 2 instead).
As we can see in Fig. 5, measures go from 0.0 (no wrong
guesses) to 0.5 (half the chance of a wrong guess). The point at
0% and 100% noise shows a high coincidence for all measures.
The interesting part of the plot is precisely the evolution from 0%
noise to 100%. We can see four lines where the measures cluster.
Setting CalB appart, the ﬁrst cluster on the bottom (with an average around 0.29 mistakes) is more robust to this misclassiﬁcationnoise and is logically composed on measures based on misclassiﬁcation or error: Acc, MFM, MAvA, MAvG and KapS. The rest of themeasures are at signiﬁcant distance (with an average around 0.31):
AUC measures, SAUC, PAUC, MSE, MPR, MAPR, MAE, CalL. The leastrobust is LogL with an average of 0.33. This means that should thedataset have noise on the class labels (i.e. noise in the test set),these latter measures will not behave well when choosing the best

6.1. Misclassiﬁcation noise
In this ﬁrst scenario, we apply noise to actual class in order tomeasure the sensitivity (or conversely, robustness) to changes (ornoise) in the actual class. Noise ranges from 0% (where no class

Fig. 5. Measure sensitivity to misclassiﬁcation noise.

36

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

model. The two main clusters are clearly consistent with the ﬁrst
column (‘‘Class Threshold”) in Table 1.
6.2. Probability noise
In this second scenario, we apply noise to the probabilities ofthe models in order to measure the sensitivity (or conversely,
robustness) to bad probability estimation. Class labels on the datasets are left unaltered, but classiﬁcation thresholds might vary.
Here, all the probabilities are modiﬁed at all the degrees ofnoise. The level of noise determines the degree to which probabilities are modiﬁed. A value a (a obtained randomly from a uniform
distribution ½b::bÞ is added to each probability. b goes from 0(noise = 0) to 0.5 (noise = 100).
As we can see in Fig. 6, the results are very different from thosein the previous scenario. Here, qualitative measures such as Acc,
MAvA, MAvG, KapS and MFM make bad choices in general. The
groups begin at the bottom with the four AUC measures, whichalong with MSE present an average wrong choice ratio between
0.087 and 0.088. The explanation of why AUC measures behave
well is simple. The ranking is only affected signiﬁcantly at great degrees of probability noise. Consequently, the good choice is preserved. The behaviour of MSE is more difﬁcult to explain, since
MSE behaves quadratically, but it can be understood if we see thatat the right of the picture, probabilities are modiﬁed +/0.25 on
average, which means that great changes, for which the quadratic
stress will make a big impact, are not common.
At quite a distance (more or less in the middle band of thegraph) we ﬁnd MPR, MAPR, MAE and PAUC (with averages about
0.13), which are all probabilistic measures equally sensitive tosmall or large probability changes. SAUC is, surprisingly, foundnext with 0.15. SAUC might be found here for different reasons, because the rankings in the middle (which have more weight due tothe inclusion of probabilities in this measure) have more relevancein this measure. At the upper band, we ﬁnd the qualitative measures (with an average value or 0.18): Acc, MAvA, MAvG, KapS
and MFM. The explanation for this bad behaviour is that for qualitative measures, if all probabilities are modiﬁed, as is the casehere, the border between classes is highly affected, and this makesthese measures worse for selecting the good model. Finally, CalL
and LogL present inverse behaviours and are more erratic thenFig. 6. Measure sensitivity to probability noise.

the rest. CalL starts badly but it is the best measure in the end. Thismeans that when distortion on probabilities is high, CalL is still
able to tell between models. LogL on the contrary, behaves reasonably well for small probability distortions but its logarithmic character makes it the worst measure for high distortions (since many
probabilities will be cut to 0 or 1, yielding 1 and 0 logarithms).
With this experiment, we have a view which gives more information than that seen in column (‘‘Calibration”) in Table 1, especially for probabilistic measures.
6.3. Ranking noise
The third scenario introduces noise to the ranking. In particular,
given a model where the probabilities are sorted, we introduce
random swappings of two consecutive elements. The degree ofnoise goes from no swapping (noise = 0) to 100.000 swappings(noise = 80).
Class labels on the datasets remain unaltered, and classiﬁcationthresholds are constant at 0.5. However, calibration, misclassiﬁcation and separability are affected by this type of noise.
The results shown on Fig. 7 indicate three types of measures. On
the one hand (excluding CalB), at the top, we have the measures
based on misclassiﬁcation or error: Acc, MFM, MAvA, MAvG and
KapS. These are more affected by these swappings than the restsince chained swappings are more frequent in probabilities around
0.5, which implies a change of class. On the other hand, we ﬁnd theprobabilistic and ranking measures such as SAUC, PAUC, MSE, MPR,
MAPR, MAE, CalL and the AUC measures. These clusters are clearlyconsistent with the third column (‘‘Ranking”) in Table 1.
6.4. Class frequency variation
The last experiment evaluates what happens if one of the classes has few examples and how this affects the robustness of a comparison between models. From the original classes 0 and 1, we
progressively eliminate examples from class 1. At a noise level of
0 we eliminate no elements from class 1. At a noise level of 50
we eliminate all the elements (50 elements) from class 1.
The results in Fig. 8 show that the AUC-based measures (thefour AUCs, and SAUC, PPAUC/MAPR) are the ones which behave
worst, because of their 1-vs-1 or 1-vs-n deﬁnition. We ﬁnd that

Fig. 7. Measure sensitivity to ranking noise.

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

SAUC and PAUC/MAPR are especially bad. At a distance, we ﬁnd
MAvA and MAvG which have a non monotonic behaviour. At ashort distance MPR/MAE. LogL, CalL and MSE are quite robust.
Then, MFM and KapS, which beahve very well until value 30. Acc
is very robust and remains almost constant. Even with noise close
to 50 where about only one element is of class 1, the selection error
is just 0.034.
The interpretation is straightforward in this case. If we want tomeasure the quality of models that might be affected by classes
with a very low percentage of elements, the AUC-based measuresand the MPR, MAE and macro-averages are not a good idea, because the global measure is heavily inﬂuenced by a poor assessment of an infrequent class. This is precisely because thesemeasures give equal value to all classes independently of their frequency. On the other hand, Acc, MFM, KapS, LogL, CalL and MSE
give a relevance to each class which is proportional to its frequency. In this sense a badly assessed class is not a problem. Thisis consistent with the fourth column (‘‘Class Frequency”) on Table
1, and the dendrograms for imbalanced datasets shown in the previous section.
6.5. Discussion
From the previous four scenarios, we can say that accuracy andother qualitative measures are the best when noise is present on
the dataset (the ﬁrst experiment). Consequently, models evaluatedwith qualitative measures will be more robust when concept drift
or other strong changes appear in the evidence. Probability-based
measures are not good here and AUC measures behave relatively
well. However, qualitative measures are very bad when distortionis produced during learning because a bad algorithm is used or
small training datasets (models are distorted, which are reproduced by the second and the third experiment). In these cases,
AUC measures are the best. According to this, if we have a learning
scenario where distortion might happen on the datasets or on thelearning process, the AUC measure is preferable, as has been shownin many previous studies (e.g. Rosset, 2004). Finally, if we havevery few examples from a class, measures which are based on
macro-averages or 1-vs-1 or 1-vs-n combinations are a bad option,
because they will be highly inﬂuenced by a bad estimation of theerror for the minority class.

37

7. Conclusions
We have studied the relationships between the most common
performance measures for classiﬁers. In this study, we have started
from the deﬁnitions, then we have designed a set of experiments toanalyse the correlations between measures and their sensitivity to
several identiﬁed traits. The results uncover the existence ofimportant similarities between measures but also signiﬁcant differences between others.
The previous analysis shows that most of the measures used inmachine learning and pattern recognition for evaluating classiﬁers
really measure different things, especially for multiclass problems
and problems with imbalanced class distribution, where correlations are worse. One of the most surprising results from the study
is that the correlations between metrics inside the same family are
not very high, showing that with a probabilistic understanding oferror, it is very different to use MSE, LogL or MPR. It is even moredifferent for the calibration measures. With a qualitative understanding of error, it is still different to use Acc or MAvG, althoughcorrelations in this group are higher. The only compact group happens when we want to rank predictions, and it is not signiﬁcantly
different to use different variants of AUC. Consequently, the previous analyses in pattern recognition or machine learning (stating,e.g., that one method is better than other) using different metrics(even inside the same family, except AUC measures) could not becomparable and extensible to the other metrics, since, the differences in performance between modern machine learning methods
are usually tight.
As future work, one interesting issue would be to analyse therelationship between measures when one is used with a small sample and the other is used with a large sample from the same distribution. This would complete our sensitivity study on whichmeasure captures more information and is more robust for small
datasets. Another line of future research would be the developmentof new measures (not as an average of measures as other works havedone (Caruana and Niculescu-Mizil, 2004; Huang and Ling, 2007),
but in the way suggested at the end of section 3.1), or the inclusionof more measures in the study, such as the chi-square statistic (Palocsay et al., 2001), or the Critical Success Index (CSI) and Heidke’s
Skill Statistic (HSS) (Marzban, 1998; Marzban and Haupt, 2005).
Summing up, apart from the clariﬁcation and the many observations found in the relationship between metrics and their sensitivity to several characteristics, this work can be used as areference when comparing two different experimental works in literature which use different metrics, in order to see whether the results are comparable or not.
Acknowledgements
The authors would like to thank David L. Dowe for some comments and discussions on LogLoss and other probabilistic measures.
We also thank the anonymous reviewers for their corrections andhelpful comments. This work has been partially supported by the
EU (FEDER) and the Spanish MEC under Grant TIN 2007-68093C02-02, Generalitat Valenciana under Grant GV06/301, UPV underGrant TAMAT and the Spanish project ‘‘Agreement Technologies”(Consolider Ingenio CSD2007-00022).
References

Fig. 8. Measure sensitivity to class frequency noise.

Baeza-Yates, R., Ribeiro-Neto, B., 1999. Modern Information Retrieval. Addison
Wesley.
Bishop, C.M., 1995. Neural Networks for Pattern Recognition. Oxford University
Press.
Blake, C.L., Merz, C.J., 1998. UCI Repository of Machine Learning Datbases.
Brier, G.W., 1950. Veriﬁcation of forecasts expressed in terms of probabilities. Mon.
Weather Rev. 78, 1–3.

38

C. Ferri et al. / Pattern Recognition Letters 30 (2009) 27–38

Buja, A., Stuetzle, W., Shen, Y., 2005. Loss functions for binary class probability
estimation: Structure and applications. <http://www-stat.wharton.upenn.edu/~buja/PAPERS/paper-proper-scoring.pdf>.
Caruana, R., Niculescu-Mizil, A., 2004. Data mining in metric space: An empiricalanalysis of supervised learning performance criteria. In: Proceedings of the 10th
ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining, KDD’2004, pp. 69–78.
Cohen, J., 1960. A coefﬁcient of agreement for nominal scales. J. Educ. Psychol. Meas.
20, 37–46.
Cortes, C., Mohri, M., 2003. AUC optimization vs. error rate minimization. In:
Advances in Neural Information Processing Systems 16, NIPS 2003.
Davis, J., Goadrich, M., 2006. The relationship between precision-recall and ROC
curves. In: ICML’06: Proceedings of the 23rd International Conference on
Machine Learning, pp. 233–240.
DeGroot, M., Fienberg, S., 1982. The comparison and evaluation of forecasters.
Statistician 31 (1), 12–22.
Domingos, P., Provost, F., 2003. Tree induction for probability-based ranking. Mach.
Learn. 52, 199–216.
Dowe, D.L., Farr, G.E., Hurst, A.J., Lentin, K.L., 1996. Information-theoretic football
tipping. In: Third Conference on Maths and Computers in Sport, vol. 14, pp.
233–241.
Esposito, F., Malerba, D., Semeraro, G., 1997. A comparative analysis of methods forpruning decision trees. IEEE Trans. Pattern Anal. Machine Intell. (TPAMI) 19 (5),476–491.
Fawcett, T., 2001. Using rule sets to maximize ROC performance. In: 2001 IEEE
International Conference on Data Mining (ICDM-01).
Fawcett, T., 2006. An introduction to ROC analysis. Pattern Recogn. Lett. 27 (8), 861–
874.
Fawcett, T., Niculescu-Mizil, A., 2007. PAV and the ROC convex hull. Mach. Learn. 68(1), 97–106.
Ferri, C., Flach, P.A., Hernández-Orallo, J., 2003. Improving the AUC of probabilistic
estimation trees. In: Machine Learning: ECML 2003, 14th European Conference
on Machine Learning, Proceedings, Lecture Notes in Computer Science, Springer,
pp. 121–132.
Ferri, C., Flach, P.A., Hernández-Orallo, J., Senad, A. 2004. Modifying ROC curves toincorporate predicted probabilities. In: Second Workshop on ROC Analysis inML.
Ferri, C., Hernández, J., Modroiu, R., 2004. An experimental comparison ofclassiﬁcation performance metrics. In: Proceedings of The Learning 2004
Conference, Universidad Miguel Hernández, Elche, Spain, pp. 39–44.
Flach, P.A., 2003. The geometry of ROC space: Understanding machine learning
metrics through ROC isometrics. In: Machine Learning, Proceedings of the 20th
International Conference (ICML 2003), pp. 194–201.
Flach, P.A., Takashi Matsubara, E., 2007. A simple lexicographic ranker andprobability estimator. In: The 18th European Conference on Machine
Learning, Springer, pp. 575–582.
Flach, P.A., Blockeel, H., Ferri, C., Hernandez-Orallo, J., Struyf, J., 2003. Decision
Support for Data Mining: Introduction to ROC Analysis and its Application.
Kluwer Academic Publishers. pp. 81–90.
Freund, Y., Iyer, R.D., Schapire, R.E., Singer, Y., 2003. An efﬁcient boosting algorithm
for combining preferences. J. Mach. Learn. Res. 4, 933–969.
Fuernkranz, J., Flach, P.A., 2005. ROC’n’ rule learning – towards a better
understanding of covering algorithms. Mach. Learn. 58 (1), 39–77.

Good, I.J., 1952. Rational decisions. J. Roy. Statist. Soc. Ser. B 14, 107–114.
Good, I.J., 1968. Corroboration, explanation, evolving probability, simplicity, and a
sharpened razor. British J. Philos. Sci. 19, 123–143.
Hand, D.J., Till, R.J., 2001. A simple generalisation of the area under the ROC curve
for multiple class classiﬁcation problems. Mach. Learn. 45 (2), 171–186.
Harrell Jr., F.E., 2001. Regression Modeling Strategies: With Applications to Linear
Models. Logistics Regression, and Survival Analysis. Springer.
Huang, J., Ling, C.X., 2005. Using AUC and accuracy in evaluating learningalgorithms. IEEE Trans. Knowl. Data Eng. (TKDE) 17 (3), 299–310.
Huang, J., Ling, C.X., 2007. Constructing new and better evaluation measures formachine learning. In: Manuela M., Veloso, (Eds.), IJCAI, pp 859–864.
Huang, J., Lu, J., Ling, C.X., 2003. Comparing naive Bayes, decision trees, and svm
with auc and accuracy. In: ICDM ’03: Proceedings of the Third IEEE International
Conference on Data Mining, Washington, DC, USA, IEEE Computer Society, p.
553.
Kuncheva, L.I., 2004. Combining Pattern Classiﬁers: Methods and Algorithms. Wiley.
Kuncheva, L.I., Whitaker, C.J., Shipp, C.A., 2003. Limits on the majority vote accuracy
in classiﬁer fusion. Pattern Anal. Appl. 6 (1), 22–31.
Lebanon, G., Lafferty, J.D., 2002. Cranking: Combining rankings using conditional
probability models on permutations. In: Machine Learning, Proceedings of the
19th International Conference (ICML 2002), pp. 363–370.
Marzban, C., 1998. Bayesian probability and scalar performance measures ingaussian models. J. Appl. Meteorol. 5, 72–82.
Marzban, C., Haupt, S.E., 2005. On genetic algorithms and discrete performancemeasures. In: Proceedings of the Fourth Conference on Artiﬁcial Intelligence
Applications to Environmental Science.
Melnik, O., Vardi, Y., Zhang, C., 2004. Mixed group ranks: Preference and conﬁdence
in classiﬁer combination. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 26 (8),973–981.
Mitchell, T.M., 1997. Machine Learning. McGraw-Hill.
Murphy, A.H., 1972. Scalar and vector partitions of the probability score: Part II. nstate situation. J. Appl. Meteorol. 11, 1182–1192.
Narasimhamurthy, A., 2005. Theoretical bounds of majority voting performance fora binary classiﬁcation problem. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)
27 (12), 1988–1995.
Palocsay, S.W., Stevens, S.P., Brookshire, R.G., 2001. An empirical evaluation ofprobability estimation with neural networks. Neural Comput. Appl. 10 (1), 48–
55.
Ripley, B.D., 1996. Pattern Recognition and Neural Networks. Cambridge University
Press, Cambridge.
Rosset, S., 2004. Model selection via the AUC. In: Machine Learning, Proceedings ofthe 21st International Conference (ICML 2004).
Sanders, F., 1963. On subjective probability forecasting. J. Appl. Meteorol. 2 (191–201).
Witten, I.H., Frank, E., 2005. Data Mining: Practical Machine Learning Tools andTechniques with Java Implementations. Elsevier.
Wu, S., Flach, P.A., Ferri, C., 2007. An improved model selection heuristic for auc. In:
18th European Conference on Machine Learning, pp. 478–489.
Young Sohn, S., 1999. Meta analysis of classiﬁcation algorithms for patternrecognition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 21 (11), 1137–1144.
Zadrozny, B., Elkan, C., 2001. Obtaining calibrated probability estimates fromdecision trees and naive bayesian classiﬁers. In: Proceedings of the 18th
International Conference on Machine Learning (ICML 2001), pp. 609–616.