AUC and its implementation in CatBoost | by Anna Volodkevich | Towards Data Science

The ROC curve shows the model’s ability to distinguishing between classes.
The model which randomly assigns a class to object is a ‘bad’ classifier and has a diagonal ROC curve. The better is the classifier, the higher is the ROC curve. The ROC curve is plotted with TPR, True Positive Rate, on the y-axis against the FPR, False Positive Rate, on the x-axis. The curve also could be interpreted in terms of Sensitivity and Specificity of the model with Sensitivity on the y-axis and (1-Specificity) on the x-axis.
Building and visualizing the ROC curve could be used to measure classification algorithm performance with different probability boundaries and select the probability boundary required to achieve the specified false-positive or false-negative rate.
AUC is the Area Under the ROC Curve. The best AUC = 1 for a model that ranks all the objects right (all objects with class 1 are assigned higher probabilities then objects of class 0). AUC for the ‘bad’ classifier which is working as random guessing is equal to 0.5.
AUC is used for binary classification, multiclass classification, and ranking problems. AUC measures the proportion of correctly ordered objects and the capability of the model to distinguish between the classes.
The AUC has an important statistical property: the AUC of a classifier is equivalent to the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance.
There are implementations of AUC in CatBoost for binary classification, multiclassification and ranking problems. You can calculate AUC during training for overfitting detection and automated best model selection, evaluate the model on new data with model.eval_metric and use AUC as a metric for predictions evaluation and comparison with utils.eval_metric. See examples of models fitting and AUC calculation with CatBoost in the section How to use AUC in Catboost on real datasets?
Read more about ROC on Wikipedia.
To get an understanding of the ROC curve building from scratch see the article.
To get an understanding of AUCμ for multiclass classification see the article.
AUC for binary classification is calculated according to the following formula:
where ai,aj — predictions (probabilities) of objects to belong to the class 1 given by the algorithm.
The sum is calculated on all pairs of objects i, j such that: ti=0, tj=1 where t is the true class label.
I is an indicator function equal to 1 if objects i, j are ordered correctly:
A user could assign weights for each object to specify the importance of each object in metric calculation.
Let’s look at a simple example of ROC curve building and AUC calculation:
We have a sample of 10 objects, 4 from class 1 and 6 from class 0 (‘class’ column). All objects are equally important (weights = 1). Assume we predicted probabilities of objects to come from class 1 (‘prediction’ column).
The denominator is equal to the number of pairs of objects i, j such that true class ti=0,tj=1. As there are 4 objects from class 1 and 6 objects from class 0, the denominator is equal to 24.
For each object from class 1 count the number of objects from class 0, which are below (has less probability) in the sorted dataset.
We add +1 of objects ordered correctly (e.g. objects with IDs 0 and 4) and +0.5 if probabilities are equal (e.g. objects with IDs 1 and 7).
For example, the object with ID = 1 adds +4.5 to AUC numerator as correctly ordered with objects with IDs: 8, 3, 6, 9 and has equal prediction with object 7:
denominator = sum(toy_example['class'] == 1) * sum(toy_example['class'] == 0)numerator = (6 + 5 + 4.5 + 2)manually_calculated_auc = numerator / denominatormanually_calculated_auc = 0.7292
Let’s calculate the same metric with Catboost and ensure everything is right:
catboost_auc = eval_metric(toy_example['class'], toy_example['prediction'], 'AUC')[0]
To build a ROC curve put the number of objects of class 1 (n1) in the square with side 1 along the y-axis, and the number of objects of class 0 (n0) along the x-axis.
Going by the list of target labels dorted by model predictions in descending order will add 1/n1 along the y-axis encountering an object of class 1 and 1/n0 along the x-axis encountering an object of class 0.
Going from top to bottom by the ‘class’ column we go up on the plot if the class is 1 or right if the class is 0. When meet objects with equal prediction add the number of 1 along y and the number of 0 along x and go by the diagonal of the rectangle.
CatBoost allows us to assign a weight to each object in the dataset for AUC calculation according to the formula above. If no weights assigned, all weights are equal to 1 and thus AUC is proportional to the number of correctly ordered pairs. If weights assigned this property is changed.
Weights are useful for unbalanced datasets. If there is a class or a group of objects with a small number of samples in the train dataset it could be reasonable to increase the weights for corresponding objects.
For example, if we assign weight 10 to one object it can be understood as adding 9 of the same objects to the dataset.
Let’s calculate AUC for an example of the wrong and right classification of one important object with weight = 10.
roc_auc = catboost.utils.eval_metric(toy_example[‘class’], toy_example[‘prediction’], ‘AUC’, weight=toy_example[‘weight’])[0]
AUC = 0.9167. An important object is correctly ordered, AUC increases
roc_auc = catboost.utils.eval_metric(toy_example['class'], toy_example['prediction'], 'AUC', weight=toy_example['weight'])[0]
AUC = 0.4417. An important object is incorrectly ordered, AUC decreases
There are two AUC metrics implemented for multiclass classification in Catboost.
The first is OneVsAll. AUC value is calculated separately for each class according to the binary classification calculation principles.
The second is AUCμ, which is designed to meet the following properties:
Property 1. If a model gives the correct label the highest probability on every example, then AUC = 1
Property 2. Random guessing on examples yields AUC = 0.5
Property 3. AUC is insensitive to class skew
AUCμ could be interpreted as an average of pairwise AUCs between the classes. Details could be found in the article.
OneVsAll AUC in Catboost returns n AUC values for n-class classification. The value is calculated separately for each class k numbered from 0 to K–1 according to the binary classification calculation rules. The objects of class k are considered positive, while all others are considered negative.
It works almost the same way as the binary classification, so I skip calculation details here (they are present in the iPython notebook).
OneVsAll AUC measures the ability of the classifier to distinguish objects of one class from another.
An advantage of OneVsAll is the ability to monitor performance on different classes separately.
AUCμ could be used as an evaluation metric to prevent overfitting, while OneVsAll AUC could be used only for information as contains k values (k is the number of classes).
Let’s have a look at the AUCμ formula:
K is a number of classes, i < j ≤ K are the classes’ numbers.
S(i, j) is a separability measure between classes i and j defined as:
where
I is indicator function applied to results of O, which returns 1 if O returns a positive value, 0 if O returns a negative value, and 0.5 if O returns 0.
O is an orientation function indicating if the two instances are ordered correctly, incorrectly, or tied. O returns a positive value if the predictions are ranked correctly, a negative value if they are ranked incorrectly, and 0 if their rank is tied.
a, b are objects from classes i, j respectively,
p(a), p(b) are vectors of predicted probabilities of the object to belong to the classes,
y(a), y(b) are true labels of objects a, b (one-hot vectors),
vi j is a two-class decision hyperplane normal vector.
O is calculated as follows:
Vectors v_i,j come from misclassification cost matrix A, which is defined manually following the learning task.
vi j=A i.−A j.
A i j is the cost of classifying an instance as class i when its true class is j. Then A defines a partition on the (K−1)−simplex and induces decision boundaries between the K classes.
Default misclassification cost matrix values are 1 everywhere, except the diagonal where they are 0. It is called the argmax partition matrix as it assigns a class with maximal probability.
Here is a partition for a 3-class classification problem with the argmax partition matrix A. For a used-defined partition matrix the boundaries (marked by arrows) are shifted.
Follow the article for more details.
CatBoost allows us to set the misclassification cost matrix A and objects’ weights to calculate AUCμ.
Let’s calculate AUCμ for a very small example with 3 classes and 4 objects:
Let’s compute S(i,j) for each ordered classes’ pair:
Let’s calculate the same metric with Catboost AUCμ and ensure everything is right:
As we can see AUCμ is pretty high, but actually, algorithm performance is bad, accuracy is 0.25. Thus it is reasonable to evaluate the other important metrics during the training.
AUCμ is a good generalizing metric to estimate algorithm ability to separate classes in multiclassification problems. See more about different approaches to multiclass AUC calculation and their properties in section 4.3 of the article.
AUCμ could be interpreted as an average of pairwise AUCs between the classes. AUCμ is a single value unlike OneVsAll AUC and thus it could be used as an evaluation metric during training for overfitting detection and trees pruning.
There are two variants of AUC for Ranking in CatBoost.
is used for models with Logloss and CrossEntropy loss functions. It has the same formula as AUC for binary classification:
ai, aj — predicted rank of objects i, j.
The sum is calculated on all pairs of objects i, j such that: ti=0, tj=1 where t is relevance.
The formula above suits for simple tasks with target values equal to 1 and 0. What if we have more target values?
If the target type is not binary, then every object i with target value t and weight w is replaced with two objects for the metric calculation:
σ1 with weight wt and target value 1
σ2 with weight w(1−t) and target value 0.
Target values must be in the range [0; 1].
Let’s calculate Classic AUC for Ranking for a small example. Assume we have 2 documents, 2 requests, corresponding relevances in the range [1, 5] and predicted relevance in the range [0, 1]:
After data preprocessing (see the notebook for details) we get:
Let’s calculate AUC:
Ranking AUC is used for ranking loss functions.
ai, aj are predicted rank of objects i, j.
The sum is calculated on all pairs of objects i, j such that: ti<tj where t is relevance.
Ranking AUC is directly measuring the quantity of correctly ranked pairs. The problem is the metric is calculated regardless of groups. In our example, we have groups (requests) and we may want to measure quality inside the groups, but currently, AUC is calculated over the dataset and AUC penalizes us if, for example, top-one document for one request is ranked lower than top-one for another document while true ranks are opposite (but still top-one for each document).
One more problem of AUC for Ranking is that it does not distinguish the top object from the others. If we have a lot of irrelevant documents, which are ranked irrelevant by the model, but bad ranking in top-10, AUC could still be high because of that “tail” of irrelevant objects, ranked lower than relevant.
AUC is a widely used metric to measure the ability of a model to distinguish classes and correctly order objects from different classes / with different relevance. AUC is not differentiable and thus cannot be used as a loss function, but it is pretty informative and useful as a metric.
We have examined the following AUC types:
AUC for binary classification. It is a classical metric, which measures the model quality. ROC curve allows selecting the probability threshold to meet required False Positive or False Negative Rates.
AUC for multiclass classification: OneVsAll and AUCμ. OneVsAll AUC helps to control algorithm performance for each class, but can not be used as a metric to prevent overfitting. On the contrary, AUCμ is good as a generalizing metric, but will not detect a problem with one class.
AUC for ranking: Classic AUC and Ranking AUC. AUC suits for ranking tasks because it is designed to measure how the algorithm distinguishes between classes and it is simply transferring to distinguishing between relevant and irrelevant objects. But AUC does not take into account that in ranking problems the first n positions are very important and treats each correctly ordered pair of documents equally.
To have a closer look at the data processing, calculation details, and examples of using AUC in CatBoost see the iPython notebook with code in CatBoost tutorials.