How To Calculate “roc_auc_score” for Regression Models
By: Samuele Mazzanti
Tags: Machine Learning, Metrics, Python, Regression, Data Science

You work as a data scientist for an auction company, and your boss asks you to build a model to predict the hammer price (i.e. the final selling price) of the items on sale. Such a model will serve two purposes:

setting a meaningful opening bid for each item; placing the most expensive items at periodic intervals during the auction. In this way, you will keep up the attention of the audience.

Since you want to predict a point value (in $), you decide to use a regression model (for instance, XGBRegressor() ). Now, how do you evaluate the performance of your model?

Let’s see Scikit’s metric toolbox for regression models:

Scikit-learn’s regression metrics [Link].

All these metrics seek to quantify how far model predictions are from the actual values. In fact, if you take a look at their formulas, you will always find this quantity:

Difference between actual value and value predicted by the model. [Figure by Author]

In other words, these metrics are great for evaluating the ability to get close to the true prices (1st objective). But they are useless for assessing the 2nd objective, which is the ability to rank the items from the most to the least expensive.

Want an example? Let’s say that the first version of your model delivers these results:

y_true = [1000.0, 2000.0, 3000.0, 4000.0, 5000.0]

y_pred = [1100.0, 1300.0, 4000.0, 4800.0, 5200.0]

Graphically,

Comparing true and predicted values in a regression problem. [Figure by Author]

If we take mean_absolute_error(y_true, y_pred) , we get 560$, which is probably not so good. However, the ranking is perfect! This means that the model is perfectly capable to discern which items will be auctioned at higher prices.

This is a very important information about our model, that we wouldn’t sense from the other regression metrics. But how could we measure the ability to rank of a regression model?

The most popular metric for assessing the ability to rank of a predictive model is roc_auc_score . So, let’s try to compute it with our data…

[Figure by Author]

We get an error!

This happens because roc_auc_score works only with classification models, either one class versus rest (“ovr”) or one versus one (“ovo”). Scikit-learn expects to find discrete classes into y_true and y_pred , while we are passing continuous values.

For this reason, we need to extend the concept of roc_auc_score to regression problems. We will call such a metric regression_roc_auc_score . In the next paragraph, we will understand how to compute it.

Looking for “regression_roc_auc_score”

Intuitively, regression_roc_auc_score shall have the following properties:

Exactly like roc_auc_score , it should be bounded between 0 (worst possible ranking) and 1 (best possible ranking), with 0.5 indicating random ranking.

, it should be bounded between 0 (worst possible ranking) and 1 (best possible ranking), with 0.5 indicating random ranking. When the target variable is binary, regression_roc_auc_score must give the same outcome of roc_auc_score (in this way, this metric will be a generalization of roc_auc_score ).

Now, how to obtain the metric we are looking for?

roc_auc_score is defined as the area under the ROC curve, which is the curve having False Positive Rate on the x-axis and True Positive Rate on the y-axis at all classification thresholds. But it’s impossible to calculate FPR and TPR for regression methods, so we cannot take this road.

Luckily for us, there is an alternative definition. In fact, according to Wikipedia, roc_auc_score coincides with “the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one”.

In other words, if we take any two observations a and b such that a > b, then roc_auc_score is equal to the probability that our model actually ranks a higher than b.

This definition is much more useful for us, because it makes sense also for regression (in fact a and b may not be restricted to be 0 or 1, they could assume any continuous value);

Moreover, calculating roc_auc_score is far easier now. In fact, it boils down to consider each possible pair of items a and b, such that a > b, and count how many times the value predicted by our model for a is actually higher than the value predicted for b (eventual ties will be counted half). Then, roc_auc_score is simply the number of successes divided by the total number of pairs.

In Python, this would be:

Python code for naive_roc_auc_score. [Code by Author]

In order to make sure that the definition provided by Wikipedia is reliable, let’s compare our function naive_roc_auc_score with the outcome of Scikit-learn.

[Figure by Author]

Great!

As said above— unlike Scikit-learn’s roc_auc_score — this version works also with continuous target variables. Let’s give it a try:

[Figure by Author]

The output is exactly what we expected. The ranking is perfect and therefore roc_auc_score equals 1.

From naive to bootstrap

Everything looks great, but the implementation above is a bit naive. In fact, naive_roc_auc_score evaluates every possible pair of observations. Thus, it requires O(n²) iterations (where n is the number of samples), and it becomes unusable as soon as n becomes a little bigger.

However, if we do not require the “exact” answer, we can obtain a good approximation through bootstrapping. Rather than evaluating every single possible pair (which would mean not less than n*(n+1)/2 pairs), we can use a number of randomly chosen pairs.

This would translate to the following Python code:

Python code for regression_roc_auc_score. [Code by Author]

regression_roc_auc_score has 3 parameters: y_true , y_pred and num_rounds . If num_rounds is an integer, it is used as the number of random pairs to consider (approximate solution). However, you can also compute the “exact” score (i.e. all possible pairs), by passing the string "exact" to num_rounds .

Show me some real data!

Are you curious to see the outcome of the function regression_roc_auc_score on a real dataset? I am too, so let’s use it on a classical dataset for regression problems: California Housing from StatLib (the data can be imported directly from Scikit-learn, which is under BSD License).

The target variable is the median house value for California districts. The dataset is made of 20,640 samples and 8 observed features.

from sklearn.datasets import fetch_california_housing

from sklearn.model_selection import train_test_split X, y = fetch_california_housing(return_X_y = True, as_frame = True)

y *= 100000 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 4321)

Now, let’s try out many different models on the training set and then compute some metrics (including regression_roc_auc_score , defined in the previous paragraph) on the test set.

from sklearn.dummy import DummyRegressor

from sklearn.neighbors import KNeighborsRegressor

from sklearn.linear_model import LinearRegression

from sklearn.svm import SVR

from sklearn.neural_network import MLPRegressor

from sklearn.tree import DecisionTreeRegressor

from sklearn.ensemble import GradientBoostingRegressor

from xgboost import XGBRegressor

from catboost import CatBoostRegressor

from sklearn.metrics import mean_absolute_error, median_absolute_error, roc_auc_score, r2_score, explained_variance_score modelnames = [

‘DummyRegressor()’,

‘KNeighborsRegressor()’,

‘LinearRegression()’,

‘SVR()’,

‘MLPRegressor(hidden_layer_sizes = (16, 16))’,

‘DecisionTreeRegressor()’,

‘GradientBoostingRegressor()’,

‘XGBRegressor()’,

‘CatBoostRegressor()’

] metricnames = [

‘mean_absolute_error’,

‘median_absolute_error’,

‘r2_score’,

‘explained_variance_score’,

‘regression_roc_auc_score’

] metrics = pd.DataFrame(index = modelnames, columns = metricnames) for modelname in modelnames:

model = eval(modelname)

pred_test = model.fit(X_train, y_train).predict(X_test)

for metricname in metricnames:

metrics.loc[modelname, metricname] = eval(f'{metricname}(y_test, pred_test)')

The following plot compares regression_roc_auc_score to mean_absolute_error for all the trained models:

Comparing regression_roc_auc_score and mean_absolute_error for different models. [Figure by Author]

As we could have been expected, the two metrics are inversely correlated. Of course, a lower mean_absolute_error tends to be associated with a higher regression_roc_auc_score .