Explicit AUC maximization. How to explicitly optimize for maximum... | by Michael Larionov, PhD | Towards Data Science

I was getting started on “IEEE-CIS Fraud Detection” Kaggle competition, and something caught my eye:
The fact that the results are evaluated based on AUC makes sense for fraud detection tasks for several reasons:
The data sets are often unbalanced, which makes it difficult to optimize for the recall or other simple metrics, unless you use oversampling or undersampling of data.In reality, the cost of false negative and false positive is different, and the actual task should include this information and optimize the actual cost, not mathematical quantities like recall or F1 score.High AUC increases the chance that we will be able to find an optimal threshold that will satisfy these requirements.
The data sets are often unbalanced, which makes it difficult to optimize for the recall or other simple metrics, unless you use oversampling or undersampling of data.
In reality, the cost of false negative and false positive is different, and the actual task should include this information and optimize the actual cost, not mathematical quantities like recall or F1 score.
High AUC increases the chance that we will be able to find an optimal threshold that will satisfy these requirements.
Usually, AUC is optimized only during hyperparameters tuning, whereas during training they use cross-entropy loss. Why not optimize AUC in the first place instead of cross-entropy?
The same question was raised in this paper:
AUC Optimization vs. Error Rate Minimization

Corinna Cortes∗and Mehryar Mohri

AT&T Labs – Research

180 Park Avenue, Florham Park, NJ 07932, USA

{corinna, mohri}@research.att.com

Abstract

The area under an ROC curve (AUC) is a criterion used in many appli-
cations to measure the quality of a classification algorithm. However,

the objective function optimized in most of these algorithms is the error

rate and not the AUC value. We give a detailed statistical analysis of the

relationship between the AUC and the error rate, including the first exact

expression of the expected value and the variance of the AUC for a fixed

error rate. Our results show that the average AUC is monotonically in-
creasing as a function of the classification accuracy, but that the standard

deviation for uneven distributions and higher error rates is noticeable.

Thus, algorithms designed to minimize the error rate may not lead to

the best possible AUC values. We show that, under certain conditions,

the global function optimized by the RankBoost algorithm is exactly the

AUC. We report the results of our experiments with RankBoost in several

datasets demonstrating the benefits of an algorithm specifically designed

to globally optimize the AUC over other existing algorithms optimizing

an approximation of the AUC or only locally optimizing the AUC.

1 Motivation

In many applications, the overall classification error rate is not the most pertinent perfor-
mance measure, criteria such as ordering or ranking seem more appropriate. Consider for

example the list of relevant documents returned by a search engine for a specific query.

That list may contain several thousand documents, but, in practice, only the top fifty or so

are examined by the user. Thus, a search engine’s ranking of the documents is more critical

than the accuracy of its classification of all documents as relevant or not. More gener-
ally, for a binary classifier assigning a real-valued score to each object, a better correlation

between output scores and the probability of correct classification is highly desirable.

A natural criterion or summary statistic often used to measure the ranking quality of a clas-
sifier is the area under an ROC curve (AUC) [8].1 However, the objective function opti-
mized by most classification algorithms is the error rate and not the AUC. Recently, several

algorithms have been proposed for maximizing the AUC value locally [4] or maximizing

some approximations of the global AUC value [9, 15], but, in general, these algorithms do

not obtain AUC values significantly better than those obtained by an algorithm designed to

minimize the error rates. Thus, it is important to determine the relationship between the

AUC values and the error rate.

∗This author’s new address is: Google Labs, 1440 Broadway, New York, NY 10018,

corinna@google.com.

1The AUC value is equivalent to the Wilcoxon-Mann-Whitney statistic [8] and closely related to

the Gini index [1]. It has been re-invented under the name of L-measure by [11], as already pointed

out by [2], and slightly modified under the name of Linear Ranking by [13, 14].

(1,1)

(0,0)

False positive rate

True positive rate

ROC Curve. AUC=0.718

True positive rate =

correctly classified positive

total positive

False positive rate =

incorrectly classified negative

total negative

Figure 1: An example of ROC curve. The line connecting (0, 0) and (1, 1), corresponding to random

classification, is drawn for reference. The true positive (negative) rate is sometimes referred to as the

sensitivity (resp. specificity) in this context.

In the following sections, we give a detailed statistical analysis of the relationship between

the AUC and the error rate, including the first exact expression of the expected value and

the variance of the AUC for a fixed error rate.2 We show that, under certain conditions, the

global function optimized by the RankBoost algorithm is exactly the AUC. We report the

results of our experiments with RankBoost in several datasets and demonstrate the benefits

of an algorithm specifically designed to globally optimize the AUC over other existing

algorithms optimizing an approximation of the AUC or only locally optimizing the AUC.

2 Definition and properties of the AUC

The Receiver Operating Characteristics (ROC) curves were originally developed in signal

detection theory [3] in connection with radio signals, and have been used since then in many

other applications, in particular for medical decision-making. Over the last few years, they

have found increased interest in the machine learning and data mining communities for

model evaluation and selection [12, 10, 4, 9, 15, 2].

The ROC curve for a binary classification problem plots the true positive rate as a function

of the false positive rate. The points of the curve are obtained by sweeping the classifica-
tion threshold from the most positive classification value to the most negative. For a fully

random classification, the ROC curve is a straight line connecting the origin to (1, 1). Any

improvement over random classification results in an ROC curve at least partially above

this straight line. Fig. (1) shows an example of ROC curve. The AUC is defined as the area

under the ROC curve and is closely related to the ranking quality of the classification as

shown more formally by Lemma 1 below.

Consider a binary classification task with m positive examples and n negative examples.

We will assume that a classifier outputs a strictly ordered list for these examples and will

denote by 1X the indicator function of a set X.

Lemma 1 ([8]) Let c be a fixed classifier. Let x1, . . . , xm be the output of c on the positive

examples and y1, . . . , yn its output on the negative examples. Then, the AUC, A, associated

to c is given by:

A =

Pm

i=1

Pn

j=1 1xi>yj

mn

(1)

that is the value of the Wilcoxon-Mann-Whitney statistic [8].

Proof. The proof is based on the observation that the AUC value is exactly the probability

P(X > Y ) where X is the random variable corresponding to the distribution of the out-
puts for the positive examples and Y the one corresponding to the negative examples [7].

The Wilcoxon-Mann-Whitney statistic is clearly the expression of that probability in the

discrete case, which proves the lemma [8].

Thus, the AUC can be viewed as a measure based on pairwise comparisons between classi-
fications of the two classes. With a perfect ranking, all positive examples are ranked higher

than the negative ones and A = 1. Any deviation from this ranking decreases the AUC.

2An attempt in that direction was made by [15], but, unfortunately, the authors’ analysis and the

result are both wrong.

m − (k − x) Positive examples

Threshold θ

k − x Positive examples x Negative examples

n − x Negative examples

Figure 2: For a fixed number of errors k, there may be x, 0 ≤ x ≤ k, false negative examples.

3 The Expected Value of the AUC

In this section, we compute exactly the expected value of the AUC over all classifications

with a fixed number of errors and compare that to the error rate.

Different classifiers may have the same error rate but different AUC values. Indeed, for a

given classification threshold θ, an arbitrary reordering of the examples with outputs more

than θ clearly does not affect the error rate but leads to different AUC values. Similarly,

one may reorder the examples with output less than θ without changing the error rate.

Assume that the number of errors k is fixed. We wish to compute the average value of the

AUC over all classifications with k errors. Our model is based on the simple assumption

that all classifications or rankings with k errors are equiprobable. One could perhaps argue

that errors are not necessarily evenly distributed, e.g., examples with very high or very low

ranks are less likely to be errors, but we cannot justify such biases in general.

For a given classification, there may be x, 0 ≤ x ≤ k, false positive examples. Since the

number of errors is fixed, there are k − x false negative examples. Figure 3 shows the cor-
responding configuration. The two regions of examples with classification outputs above

and below the threshold are separated by a vertical line. For a given x, the computation of

the AUC, A, as given by Eq. (1) can be divided into the following three parts:

A =

A1 + A2 + A3

mn

, with (2)

A1 = the sum over all pairs (xi

, yj) with xi and yj in distinct regions;

A2 = the sum over all pairs (xi

, yj) with xi and yj in the region above the threshold;

A3 = the sum over all pairs (xi

, yj) with xi and yj in the region below the threshold.

The first term, A1, is easy to compute. Since there are (m − (k − x)) positive examples

above the threshold and n − x negative examples below the threshold, A1 is given by:

A1 = (m − (k − x))(n − x) (3)

To compute A2, we can assign to each negative example above the threshold a position

based on its classification rank. Let position one be the first position above the threshold

and let α1 < . . . < αx denote the positions in increasing order of the x negative examples

in the region above the threshold. The total number of examples classified as positive is

N = m − (k − x) + x. Thus, by definition of A2,

A2 =

Xx

i=1

(N − αi) − (x − i) (4)

where the first term N − αi represents the number of examples ranked higher than the ith

example and the second term x − i discounts the number of negative examples incorrectly

ranked higher than the ith example. Similarly, let α

0

1 < . . . < α0

k−x

denote the positions of

the k − x positive examples below the threshold, counting positions in reverse by starting

from the threshold. Then, A3 is given by:

A3 =

xX0

j=1

(N

0 − α

0

j

) − (x

0 − j) (5)

with N0 = n − x + (k − x) and x

0 = k − x. Combining the expressions of A1, A2, and

A3 leads to:

A =

A1 + A2 + A3

mn

= 1 +

(k − 2x)

2 + k

2mn

−

(

Px

i=1 αi +

Px

0

j=1 α

0

j

)

mn

(6)

Lemma 2 For a fixed x, the average value of the AUC A is given by:

< A >x= 1 −

x

n + k−x

m

2

(7)

Proof. The proof is based on the computation of the average values of Px

i=1 αi and

Px

0

j=1 α

0

j

for a given x. We start by computing the average value < αi >x for a given

i, 1 ≤ i ≤ x. Consider all the possible positions for α1 . . . αi−1 and αi+1 . . . αx, when the

value of αi

is fixed at say αi = l. We have i ≤ l ≤ N − (x − i) since there need to be at

least i−1 positions before αi and N −(x−i) above. There are l −1 possible positions for

α1 . . . αi−1 and N − l possible positions for αi+1 . . . αx. Since the total number of ways

of choosing the x positions for α1 . . . αx out of N is

N

x



, the average value < αi >x is:

< αi >x=

PN−(x−i)

l=i

l

l−1

i−1



N−l

x−i



N

x

 (8)

Thus,

<

Xx

i=1

αi >x=

Px

i=1

PN−(x−i)

l=i

l

l−1

i−1



N−l

x−i



N

x

 =

PN

l=1

l

Px

i=1

l−1

i−1



N−l

x−i



N

x

 (9)

Using the classical identity: P

p1+p2=p

u

p1



v

p2



=

u+v

p



, we can write:

<

Xx

i=1

αi >x=

PN

l=1 l

N−1

x−1



N

x

 =

N(N + 1)

2

N−1

x−1



N

x

 =

x(N + 1)

2

(10)

Similarly, we have:

<

xX0

j=1

α

0

j >x=

x

0

(N0 + 1)

2

(11)

Replacing <

Px

i=1 αi >x and <

Px

0

j=1 α

0

j >x in Eq. (6) by the expressions given by

Eq. (10) and Eq. (11) leads to:

< A >x= 1 +

(k − 2x)

2 + k − x(N + 1) − x

0

(N0 + 1)

2mn

= 1 −

x

n + k−x

m

2

(12)

which ends the proof of the lemma.

Note that Eq. (7) shows that the average AUC value for a given x is simply one minus the

average of the accuracy rates for the positive and negative classes.

Proposition 1 Assume that a binary classification task with m positive examples and n

negative examples is given. Then, the expected value of the AUC A over all classifications

with k errors is given by:

< A >= 1 −

k

m + n

−

(n − m)

2

(m + n + 1)

4mn



k

m + n

−

Pk−1

x=0

m+n

x



Pk

x=0

m+n+1

x



!

(13)

Proof. Lemma 2 gives the average value of the AUC for a fixed value of x. To compute

the average over all possible values of x, we need to weight the expression of Eq. (7) with

the total number of possible classifications for a given x. There are

N

x



possible ways of

choosing the positions of the x misclassified negative examples, and similarly

N0

x0



possible

ways of choosing the positions of the x

0 = k − x misclassified positive examples. Thus, in

view of Lemma 2, the average AUC is given by:

< A >=

Pk

x=0

N

x



N0

x0



(1 −

x

n + k−x

m

2

)

Pk

x=0

N

x



N0

x0

 (14)

The problem is summarized well in the paper’s intro:
What if we did use the AUC as the optimization function instead of cross-entropy? The AUC curve can be computed using Wilcoxon-Mann-Whitney statistic:
If x and y are probabilities, the metric penalizes any occurrence where the probability of the actual positive case is less than the probability of the negative case. This is not a conventional optimization function, because it lacks additive property of MSE or cross-entropy cost functions. As it was noted in the paper,
The AUC in the equation above is not a smooth function of probabilities, because it is constructed using step functions. We can try to smooth it out, to make sure the function is always differentiable, so that we can use the conventional optimization algorithms. One of the ways to get around of this, as demonstrated in the paper, is the RankBoost loss function:
Here x and y are probabilities for the positive and negative class respectively. For example, in the logistic regression model both x and y can be expressed as sigmoid functions:
Here ξ is a predictor variable(s).
I have to admit that these models are nothing new. They are called pairwise comparison models and pioneered by Thurstone in the 1920s. For more information see this book. The foundation of this analysis is Bradley-Terry-Luce model, but the most well-known variation is Elo algorithm. The model itself is very simple and is derived by maximizing the likelihood function:
In our context, M_ij is 1 if i is from the positive class and j is from the negative class. In all other cases, it is 0. The ability values α here are logits, not the probabilities. This makes the model simpler than RankBoost, and also allows us to use it as the final layer of a neural network (in which case the values α will be just the activations of the previous layer). The loss function is:
Where i and j are positive and negative examples respectively.
We will demonstrate this approach using a simple classification problem. We will use TensorFlow to take advantage of automatic differentiation.
As you see, this is a binary classification problem using synthetic data. Instead of trying to correctly classify the results, we will try to maximize AUC.
Epoch 0 Cost = -0.12107406 Epoch 100 Cost = -0.24906836 Epoch 200 Cost = -0.2497198 Epoch 300 Cost = -0.24984348 Epoch 400 Cost = -0.24989356
Now we plot our predictions:
The plot below is based on the threshold 0.945. If I were to choose the conventional threshold 0.5 we would get all observations in one class. Why is that? The reason is that AUC is not sensitive to the bias value. Indeed, since in this model we subtracting pairwise activations, the bias value cancels out. We can add additional term to the cost function to bring the bias closer to 0.5. Or choose to do what we have done above, by hand-picking the threshold that works for us. I want to remind you, that we are optimizing for AUC, so if all observations are predicted to fall into a single class, it is cool with us as long as the AUC has the optimal value.
One can wonder if there is any usefulness in this approach for any real machine learning projects, other than data science competitions that use AUC as the evaluation criterion. But I would argue that it can be useful in case of unbalanced data set as the first step in machine learning. The second step would be to learn the bias parameter and generally fine-tuning the model by using a business-based optimization metric.
One disadvantage of the AUC cost function is that it is using pairwise comparison between observation, so the complexity of the function grows as a square of the number of observations, which could become a performance bottleneck as the number of observations grows. Also one has to come up with a good batching strategy that correctly estimates AUC based on a number of smaller samples (batches).
All code is available in my github repository. Also find more information about RankBoost below.