“roc_auc_score” Can Be Calculated Also for Regression Models | by Samuele Mazzanti | Towards Data Science

You work as a data scientist for an auction company, and your boss asks you to build a model to predict the hammer price (i.e. the final selling price) of the items on sale. Such a model will serve two purposes:
setting a meaningful opening bid for each item;placing the most expensive items at periodic intervals during the auction. In this way, you will keep up the attention of the audience.
setting a meaningful opening bid for each item;
placing the most expensive items at periodic intervals during the auction. In this way, you will keep up the attention of the audience.
Since you want to predict a point value (in $), you decide to use a regression model (for instance, XGBRegressor()). Now, how do you evaluate the performance of your model?
Let’s see Scikit’s metric toolbox for regression models:
All these metrics seek to quantify how far model predictions are from the actual values. In fact, if you take a look at their formulas, you will always find this quantity:
In other words, these metrics are great for evaluating the ability to get close to the true prices (1st objective). But they are useless for assessing the 2nd objective, which is the ability to rank the items from the most to the least expensive.
Want an example? Let’s say that the first version of your model delivers these results:
y_true = [1000.0, 2000.0, 3000.0, 4000.0, 5000.0]y_pred = [1100.0, 1300.0, 4000.0, 4800.0, 5200.0]
Graphically,
If we take mean_absolute_error(y_true, y_pred), we get 560$, which is probably not so good. However, the ranking is perfect! This means that the model is perfectly capable to discern which items will be auctioned at higher prices.
This is a very important information about our model, that we wouldn’t sense from the other regression metrics. But how could we measure the ability to rank of a regression model?
The most popular metric for assessing the ability to rank of a predictive model is roc_auc_score. So, let’s try to compute it with our data...
We get an error!
This happens because roc_auc_score works only with classification models, either one class versus rest (“ovr”) or one versus one (“ovo”). Scikit-learn expects to find discrete classes into y_true and y_pred, while we are passing continuous values.
For this reason, we need to extend the concept of roc_auc_score to regression problems. We will call such a metric regression_roc_auc_score. In the next paragraph, we will understand how to compute it.
Intuitively, regression_roc_auc_score shall have the following properties:
Exactly like roc_auc_score, it should be bounded between 0 (worst possible ranking) and 1 (best possible ranking), with 0.5 indicating random ranking.
When the target variable is binary, regression_roc_auc_score must give the same outcome of roc_auc_score (in this way, this metric will be a generalization of roc_auc_score).
Now, how to obtain the metric we are looking for?
roc_auc_score is defined as the area under the ROC curve, which is the curve having False Positive Rate on the x-axis and True Positive Rate on the y-axis at all classification thresholds. But it’s impossible to calculate FPR and TPR for regression methods, so we cannot take this road.
Luckily for us, there is an alternative definition. In fact, according to Wikipedia, roc_auc_score coincides with “the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one”.
In other words, if we take any two observations a and b such that a > b, then roc_auc_score is equal to the probability that our model actually ranks a higher than b.
This definition is much more useful for us, because it makes sense also for regression (in fact a and b may not be restricted to be 0 or 1, they could assume any continuous value);
Moreover, calculating roc_auc_score is far easier now. In fact, it boils down to consider each possible pair of items a and b, such that a > b, and count how many times the value predicted by our model for a is actually higher than the value predicted for b (eventual ties will be counted half). Then, roc_auc_score is simply the number of successes divided by the total number of pairs.
In Python, this would be:
In order to make sure that the definition provided by Wikipedia is reliable, let’s compare our function naive_roc_auc_score with the outcome of Scikit-learn.
Great!
As said above— unlike Scikit-learn’s roc_auc_score— this version works also with continuous target variables. Let’s give it a try:
The output is exactly what we expected. The ranking is perfect and therefore roc_auc_score equals 1.
Everything looks great, but the implementation above is a bit naive. In fact, naive_roc_auc_score evaluates every possible pair of observations. Thus, it requires O(n2) iterations (where n is the number of samples), and it becomes unusable as soon as n becomes a little bigger.
However, if we do not require the “exact” answer, we can obtain a good approximation through bootstrapping. Rather than evaluating every single possible pair (which would mean not less than n*(n+1)/2 pairs), we can use a number of randomly chosen pairs.
This would translate to the following Python code:
regression_roc_auc_score has 3 parameters: y_true, y_pred and num_rounds. If num_rounds is an integer, it is used as the number of random pairs to consider (approximate solution). However, you can also compute the “exact” score (i.e. all possible pairs), by passing the string "exact" to num_rounds.
Are you curious to see the outcome of the function regression_roc_auc_score on a real dataset? I am too, so let’s use it on a classical dataset for regression problems: California Housing from StatLib (the data can be imported directly from Scikit-learn, which is under BSD License).
The target variable is the median house value for California districts. The dataset is made of 20,640 samples and 8 observed features.
from sklearn.datasets import fetch_california_housingfrom sklearn.model_selection import train_test_splitX, y = fetch_california_housing(return_X_y = True, as_frame = True)y *= 100000X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 4321)
Now, let’s try out many different models on the training set and then compute some metrics (including regression_roc_auc_score, defined in the previous paragraph) on the test set.
from sklearn.dummy import DummyRegressorfrom sklearn.neighbors import KNeighborsRegressorfrom sklearn.linear_model import LinearRegressionfrom sklearn.svm import SVRfrom sklearn.neural_network import MLPRegressorfrom sklearn.tree import DecisionTreeRegressorfrom sklearn.ensemble import GradientBoostingRegressorfrom xgboost import XGBRegressorfrom catboost import CatBoostRegressorfrom sklearn.metrics import mean_absolute_error, median_absolute_error, roc_auc_score, r2_score, explained_variance_scoremodelnames = [ ‘DummyRegressor()’,  ‘KNeighborsRegressor()’, ‘LinearRegression()’,  ‘SVR()’, ‘MLPRegressor(hidden_layer_sizes = (16, 16))’, ‘DecisionTreeRegressor()’,  ‘GradientBoostingRegressor()’, ‘XGBRegressor()’, ‘CatBoostRegressor()’]metricnames = [ ‘mean_absolute_error’,  ‘median_absolute_error’, ‘r2_score’, ‘explained_variance_score’, ‘regression_roc_auc_score’]metrics = pd.DataFrame(index = modelnames, columns = metricnames)for modelname in modelnames:  model = eval(modelname)  pred_test = model.fit(X_train, y_train).predict(X_test)  for metricname in metricnames:    metrics.loc[modelname, metricname] = eval(f'{metricname}(y_test, pred_test)')
The following plot compares regression_roc_auc_score to mean_absolute_error for all the trained models:
As we could have been expected, the two metrics are inversely correlated. Of course, a lower mean_absolute_error tends to be associated with a higher regression_roc_auc_score .
The latter metric provides additional knowledge about the model performance: after calculating regression_roc_auc_score we can say that the probability that Catboost estimates a higher value for a compared to b, given that a > b, is close to 90%.
Thank you for reading! I hope you enjoyed this article.
I appreciate feedback and constructive criticism. If you want to talk about this article or other related topics, you can text me at my Linkedin contact.